Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s22501 (P0 jython_slave PY3P0) in workspace /data/workspace/temp_win_private_job_3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF > /usr/bin/git init /data/workspace/temp_win_private_job_3 # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 5211d4c425c2693f9ed4de320e82d1c6b0873f86 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 5211d4c425c2693f9ed4de320e82d1c6b0873f86 > /usr/bin/git rev-list 5211d4c425c2693f9ed4de320e82d1c6b0873f86 # timeout=10 [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins8373584077483475821.sh + echo '[global] username:Administrator password:Membase123 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.107.126 port:8091 [_2] ip:172.23.107.127 port:8091 [_3] ip:172.23.107.129 port:8091 [_4] ip:172.23.107.131 port:8091 [_5] ip:172.23.107.142 port:8091' [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins6382211605629233606.sh + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 5 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_win_private_job_3/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 12s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_252 (Oracle Corporation 25.252-b09) OS: Linux 3.10.0-1127.13.1.el7.x86_64 amd64 + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.7 -P client_version=default -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.7 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/ssh.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/eagles_all_around.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main scripts/install.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: conf/collections/collections_failover_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_failover_crud_on_collections'} Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 1, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:18:23,361 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:18:24,331 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:18:27,602 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0xd22b6a3d, 0.0.0.0/0.0.0.0:37310] timeout=300.0>) 2021-05-17 00:18:49,316 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xc8e444c8, 0.0.0.0/0.0.0.0:59249] timeout=300.0>) 2021-05-17 00:18:51,357 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x0a25d28f, 0.0.0.0/0.0.0.0:52035] timeout=300.0>) 2021-05-17 00:18:53,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xc5a796ed, 0.0.0.0/0.0.0.0:33020] timeout=300.0>) 2021-05-17 00:19:17,285 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xeab81305, 0.0.0.0/0.0.0.0:44047] timeout=300.0>) 2021-05-17 00:19:19,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x7bc34bf8, 0.0.0.0/0.0.0.0:60958] timeout=300.0>) 2021-05-17 00:19:21,355 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x91416262, 0.0.0.0/0.0.0.0:43846] timeout=300.0>) 2021-05-17 00:19:23,382 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0xa200be5a, 0.0.0.0/0.0.0.0:46957] timeout=300.0>) 2021-05-17 00:19:25,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xcd21a2fd, 0.0.0.0/0.0.0.0:47241] timeout=300.0>) 2021-05-17 00:19:50,178 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x40fcbeb6, 0.0.0.0/0.0.0.0:34267] timeout=300.0>) 2021-05-17 00:19:52,206 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x8d1c475a, 0.0.0.0/0.0.0.0:47931] timeout=300.0>) 2021-05-17 00:19:54,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x6ad961f2, 0.0.0.0/0.0.0.0:51299] timeout=300.0>) 2021-05-17 00:19:56,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x97148c38, 0.0.0.0/0.0.0.0:56528] timeout=300.0>) 2021-05-17 00:19:58,313 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x84c9651f, 0.0.0.0/0.0.0.0:60804] timeout=300.0>) 2021-05-17 00:20:23,196 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x8a8aa2e7, 0.0.0.0/0.0.0.0:45262] timeout=300.0>) 2021-05-17 00:20:25,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x6fc702da, 0.0.0.0/0.0.0.0:57773] timeout=300.0>) 2021-05-17 00:20:27,255 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x32dcb4ff, 0.0.0.0/0.0.0.0:48925] timeout=300.0>) 2021-05-17 00:20:29,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x39368e22, 0.0.0.0/0.0.0.0:46092] timeout=300.0>) 2021-05-17 00:20:31,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xfd46a090, 0.0.0.0/0.0.0.0:52959] timeout=300.0>) 2021-05-17 00:20:55,869 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x620cafeb, 0.0.0.0/0.0.0.0:53061] timeout=300.0>) 2021-05-17 00:20:57,895 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xa4aad489, 0.0.0.0/0.0.0.0:58083] timeout=300.0>) 2021-05-17 00:20:59,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xa1fdc501, 0.0.0.0/0.0.0.0:39363] timeout=300.0>) 2021-05-17 00:21:01,948 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x44123ef1, 0.0.0.0/0.0.0.0:60660] timeout=300.0>) 2021-05-17 00:21:03,976 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:21:22,219 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:21:36,270 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:21:36,309 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:21:36,369 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:21:52,003 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.56833333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 00:21:52,046 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:22:02,173 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:22:02,216 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2150001526 sec 2021-05-17 00:22:02,625 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 00:22:49,197 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 00:22:49,601 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 993609288 | 107565129 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 301356080 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1288556936 | 51935309 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 00:22:49,602 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 00:23:41,921 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:23:41,963 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:24:07,217 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..............................***......................................................................*X | 121.151 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:24:09,006 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....***.................................................................................**.............X | 25614.782 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:24:09,823 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........................**........................................................................**....*X | 135.670 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:24:10,473 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9727803202 | 6437847040 | 4270555136 | 2563874816 / 7511588864 | 100098 / 271488 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.663153682 | 6437847040 | 4244692992 | 2576601088 / 7511588864 | 98760 / 255949 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8435835996 | 6437847040 | 4263268352 | 2580705280 / 7511588864 | 97692 / 284744 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.7930055922 | 6437847040 | 3888656384 | 3154681856 / 9530753024 | 99763 / 258387 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8951677316 | 6437847040 | 4261179392 | 2571583488 / 7511588864 | 98812 / 255994 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 00:24:23,805 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:25:51,835 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 213250248 | 296259731 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 318381232 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 848323416 | 628598626 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 00:25:51,836 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:26:02,401 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 00:26:04,026 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 00:26:04,028 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_rebalance_out 2021-05-17 00:26:04,035 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:26:13,194 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:26:13,463 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 00:26:13,463 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 00:26:30,509 | test | INFO | pool-2-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................................................................................X | 19863.163 | +-------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:26:30,846 | test | INFO | pool-2-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 6139.082 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:27:40,743 | test | INFO | pool-2-thread-29 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 86.7849998474 seconds to complete 2021-05-17 00:27:40,750 | test | INFO | pool-2-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:29:45,819 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0629999637604 seconds 2021-05-17 00:30:34,052 | test | INFO | pool-2-thread-17 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 48.0609998703 seconds to complete 2021-05-17 00:30:34,059 | test | INFO | pool-2-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:32:39,121 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 00:32:39,305 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 00:32:39,308 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:32:45,446 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:33:00,361 | test | INFO | pool-2-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5044.908 | | 4932.970 | ................................................................................................X | 4932.970 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:33:00,828 | test | INFO | pool-2-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................................*................................................X | 21149.525 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:33:01,865 | test | INFO | pool-2-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 6100.229 | | 2389.509 | ......................................*X | 2389.509 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:33:02,145 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,147 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,197 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,197 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:33:18,664 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.456742790465 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 79.9473350888 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 70.5028081486 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 58.4394373708 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.363333333333 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 00:33:18,687 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:33:23,759 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 4.94 2021-05-17 00:33:28,838 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 11.41 2021-05-17 00:33:33,914 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 17.72 2021-05-17 00:33:39,002 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 27.4 2021-05-17 00:33:44,525 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 00:33:49,608 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 42.8 2021-05-17 00:33:54,680 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 50.46 2021-05-17 00:33:59,749 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 62.46 2021-05-17 00:34:04,826 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 00:34:09,926 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 76.03 2021-05-17 00:34:15,013 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 81.03 2021-05-17 00:34:20,101 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 88.16 2021-05-17 00:34:25,250 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 95.36 2021-05-17 00:34:35,351 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:34:35,506 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 76.8420000076 sec 2021-05-17 00:34:43,563 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:35:30,158 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 00:35:32,105 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 00:35:33,043 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 00:35:33,045 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 00:35:45,290 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 00:35:45,292 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 00:36:04,190 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 00:36:04,191 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 00:37:02,770 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.135999918 seconds to complete 2021-05-17 00:37:02,790 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1130.905s During the test, OK Remote Connections: 172, Disconnections: 172 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:37:12,240 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:37:12,937 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:37:13,404 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff5 type=client open_count=1 channel=[id: 0xbc9b8c5c, 0.0.0.0/0.0.0.0:54272] timeout=300.0>) 2021-05-17 00:37:34,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff6 type=client open_count=1 channel=[id: 0x558e8f35, 0.0.0.0/0.0.0.0:52217] timeout=300.0>) 2021-05-17 00:37:36,434 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff7 type=client open_count=1 channel=[id: 0x2072cf39, 0.0.0.0/0.0.0.0:45094] timeout=300.0>) 2021-05-17 00:37:38,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ff8 type=client open_count=1 channel=[id: 0x0dcede3e, 0.0.0.0/0.0.0.0:52625] timeout=300.0>) 2021-05-17 00:38:02,874 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ff9 type=client open_count=1 channel=[id: 0x33e45d13, 0.0.0.0/0.0.0.0:33161] timeout=300.0>) 2021-05-17 00:38:04,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ffa type=client open_count=1 channel=[id: 0x7bdac34b, 0.0.0.0/0.0.0.0:52880] timeout=300.0>) 2021-05-17 00:38:06,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffb type=client open_count=1 channel=[id: 0x9e140f7a, 0.0.0.0/0.0.0.0:56606] timeout=300.0>) 2021-05-17 00:38:30,625 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffc type=client open_count=1 channel=[id: 0xcdadd421, 0.0.0.0/0.0.0.0:47974] timeout=300.0>) 2021-05-17 00:38:32,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffd type=client open_count=1 channel=[id: 0x98f07702, 0.0.0.0/0.0.0.0:39940] timeout=300.0>) 2021-05-17 00:38:34,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x99ffe type=client open_count=1 channel=[id: 0x6850fc93, 0.0.0.0/0.0.0.0:58029] timeout=300.0>) 2021-05-17 00:38:58,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x99fff type=client open_count=1 channel=[id: 0x6a2f6f47, 0.0.0.0/0.0.0.0:50821] timeout=300.0>) 2021-05-17 00:39:00,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x9a000 type=client open_count=1 channel=[id: 0x4af76def, 0.0.0.0/0.0.0.0:48388] timeout=300.0>) 2021-05-17 00:39:02,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a001 type=client open_count=1 channel=[id: 0x370fa448, 0.0.0.0/0.0.0.0:37162] timeout=300.0>) 2021-05-17 00:39:27,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a002 type=client open_count=1 channel=[id: 0xbc5635af, 0.0.0.0/0.0.0.0:45831] timeout=300.0>) 2021-05-17 00:39:29,364 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a003 type=client open_count=1 channel=[id: 0xbacf31e3, 0.0.0.0/0.0.0.0:43070] timeout=300.0>) 2021-05-17 00:39:31,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:39:48,543 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:40:02,201 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:40:02,226 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:40:02,266 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:40:16,194 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.365 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+-------+--------------+ 2021-05-17 00:40:16,220 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:40:26,296 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:40:26,323 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.129999876 sec 2021-05-17 00:40:26,578 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 00:41:14,964 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 00:41:15,384 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 865238056 | 120364342 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 291175888 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1126643576 | 83916269 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 00:41:15,387 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 00:41:58,657 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:41:58,670 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:42:19,477 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 169.413 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:42:20,063 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................................................**..................................................X | 26831.559 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:42:21,786 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 197.899 | | 92.226 | .............................................*X | 92.226 | +--------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:42:22,088 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.3200113331 | 6437847040 | 4303294464 | 2529480704 / 7511588864 | 82644 / 160843 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.5563973484 | 6437847040 | 4229492736 | 2583810048 / 7511588864 | 81474 / 246256 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.5309317931 | 6437847040 | 4256976896 | 2560475136 / 7511588864 | 80890 / 203264 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.532414803 | 6437847040 | 3915497472 | 3090124800 / 9530753024 | 82449 / 141853 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.3730564751 | 6437847040 | 4276359168 | 2538450944 / 7511588864 | 82311 / 208916 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 00:42:36,957 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:44:06,842 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212748328 | 298433305 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319770704 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 832897704 | 637498477 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 00:44:06,845 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:44:17,026 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 00:44:17,026 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 00:44:17,028 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 00:45:16,487 | test | INFO | pool-4-thread-27 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.2430000305 seconds to complete 2021-05-17 00:45:16,493 | test | INFO | pool-4-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:47:21,559 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0620000362396 seconds 2021-05-17 00:48:12,648 | test | INFO | pool-4-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 50.9049999714 seconds to complete 2021-05-17 00:48:12,654 | test | INFO | pool-4-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:50:17,717 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0590000152588 seconds 2021-05-17 00:50:17,719 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 00:50:17,743 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:50:24,059 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:50:46,012 | test | INFO | pool-4-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................**X | 3762.955 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:50:47,645 | test | INFO | pool-4-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****............................................................................***........................X | 17005.162 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:50:49,187 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****................................*....................................................................*X | 2549.964 | | 2266.872 | .......................................................................................X | 2266.872 | | 1612.310 | ..............................................................X | 1612.310 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:50:49,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,582 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,582 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:51:03,928 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:51:04,059 | test | INFO | pool-4-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.624989583507 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 80.835814869 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 76.7170547158 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 55.3642560709 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.236662722288 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 00:51:04,088 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:51:09,177 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:51:10,825 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:51:14,584 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.76 2021-05-17 00:51:19,918 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.86 2021-05-17 00:51:25,168 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 1.67 2021-05-17 00:51:29,894 | test | INFO | pool-4-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........................................................................................**...........X | 2561.451 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:51:30,306 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 3.14 2021-05-17 00:51:31,786 | test | INFO | pool-4-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......................X | 5168.490 | | 5168.490 | ......................*...................................................*..........................X | 22099.988 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:51:33,701 | test | INFO | pool-4-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...........*.......*..........................*........................................................*X | 4827.747 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:51:35,390 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 5.8 2021-05-17 00:51:40,476 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 9.01 2021-05-17 00:51:45,561 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.48 2021-05-17 00:51:50,647 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.21 2021-05-17 00:51:55,726 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 22.63 2021-05-17 00:52:00,819 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 28.49 2021-05-17 00:52:06,974 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 00:52:12,061 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.64 2021-05-17 00:52:17,150 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 48.01 2021-05-17 00:52:22,256 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 56.18 2021-05-17 00:52:27,342 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 64.64 2021-05-17 00:52:32,418 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.24 2021-05-17 00:52:37,502 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 76.09 2021-05-17 00:52:42,621 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.85 2021-05-17 00:52:47,727 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 87.82 2021-05-17 00:52:52,812 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 93.73 2021-05-17 00:52:57,888 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 99.39 2021-05-17 00:53:08,009 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:53:08,164 | test | INFO | pool-4-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 124.105999947 sec 2021-05-17 00:53:17,802 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:54:07,140 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 00:54:07,141 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 00:54:19,278 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 98% 2021-05-17 00:54:19,279 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 98% 2021-05-17 00:54:33,747 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 00:54:33,749 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 00:55:32,213 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1349999905 seconds to complete 2021-05-17 00:55:32,230 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1109.395s OK During the test, Remote Connections: 172, Disconnections: 172 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 3, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:55:41,701 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:55:42,367 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:55:42,798 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325c8 type=client open_count=1 channel=[id: 0x02d92d3c, 0.0.0.0/0.0.0.0:55453] timeout=300.0>) 2021-05-17 00:56:03,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325c9 type=client open_count=1 channel=[id: 0x16a1c393, 0.0.0.0/0.0.0.0:41002] timeout=300.0>) 2021-05-17 00:56:05,798 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325ca type=client open_count=1 channel=[id: 0xbb1a638d, 0.0.0.0/0.0.0.0:57178] timeout=300.0>) 2021-05-17 00:56:07,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325cb type=client open_count=1 channel=[id: 0xcdb74187, 0.0.0.0/0.0.0.0:46543] timeout=300.0>) 2021-05-17 00:56:09,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325cc type=client open_count=1 channel=[id: 0x4e9edf2e, 0.0.0.0/0.0.0.0:39153] timeout=300.0>) 2021-05-17 00:56:11,857 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325cd type=client open_count=1 channel=[id: 0x7d0308a4, 0.0.0.0/0.0.0.0:46836] timeout=300.0>) 2021-05-17 00:56:38,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325ce type=client open_count=1 channel=[id: 0x194d6629, 0.0.0.0/0.0.0.0:51793] timeout=300.0>) 2021-05-17 00:56:40,346 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325cf type=client open_count=1 channel=[id: 0x6f242d88, 0.0.0.0/0.0.0.0:55222] timeout=300.0>) 2021-05-17 00:56:42,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325d0 type=client open_count=1 channel=[id: 0x9b3aaa75, 0.0.0.0/0.0.0.0:48828] timeout=300.0>) 2021-05-17 00:56:44,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d1 type=client open_count=1 channel=[id: 0x11dca817, 0.0.0.0/0.0.0.0:37305] timeout=300.0>) 2021-05-17 00:57:08,474 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d2 type=client open_count=1 channel=[id: 0x5964fad3, 0.0.0.0/0.0.0.0:43020] timeout=300.0>) 2021-05-17 00:57:10,493 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d3 type=client open_count=1 channel=[id: 0x99032c2a, 0.0.0.0/0.0.0.0:33179] timeout=300.0>) 2021-05-17 00:57:12,512 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d4 type=client open_count=1 channel=[id: 0x8edf7874, 0.0.0.0/0.0.0.0:53068] timeout=300.0>) 2021-05-17 00:57:14,533 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d5 type=client open_count=1 channel=[id: 0x357232ce, 0.0.0.0/0.0.0.0:34306] timeout=300.0>) 2021-05-17 00:57:38,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d6 type=client open_count=1 channel=[id: 0xa273d954, 0.0.0.0/0.0.0.0:37356] timeout=300.0>) 2021-05-17 00:57:40,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d7 type=client open_count=1 channel=[id: 0x3828793b, 0.0.0.0/0.0.0.0:36633] timeout=300.0>) 2021-05-17 00:57:42,280 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325d8 type=client open_count=1 channel=[id: 0x84e3337b, 0.0.0.0/0.0.0.0:57379] timeout=300.0>) 2021-05-17 00:58:06,551 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325d9 type=client open_count=1 channel=[id: 0xbcae4a5c, 0.0.0.0/0.0.0.0:48979] timeout=300.0>) 2021-05-17 00:58:08,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325da type=client open_count=1 channel=[id: 0x0e1b016f, 0.0.0.0/0.0.0.0:43883] timeout=300.0>) 2021-05-17 00:58:10,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325db type=client open_count=1 channel=[id: 0x8690fc52, 0.0.0.0/0.0.0.0:53603] timeout=300.0>) 2021-05-17 00:58:12,607 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325dc type=client open_count=1 channel=[id: 0xbb1605a2, 0.0.0.0/0.0.0.0:55398] timeout=300.0>) 2021-05-17 00:58:14,627 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325dd type=client open_count=1 channel=[id: 0xfd39469e, 0.0.0.0/0.0.0.0:43569] timeout=300.0>) 2021-05-17 00:58:16,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:58:33,713 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:58:47,446 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:58:47,467 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:58:47,510 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:59:00,839 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.51660805653 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 00:59:00,875 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:59:10,950 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:59:10,976 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1380000114 sec 2021-05-17 00:59:11,236 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:00:10,711 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:00:10,982 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 774107752 | 157838792 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 290193472 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 868677800 | 111247271 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:00:10,983 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:00:55,142 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:00:55,154 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:01:17,453 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 129.747 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:01:17,707 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....................................................*...............................................*X | 108.356 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:01:19,117 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..................................................................................................**..X | 23952.315 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:01:19,407 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.3540586647 | 6437847040 | 4138688512 | 2606309376 / 7511588864 | 96935 / 293317 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.846435825 | 6437847040 | 4145115136 | 2598170624 / 7511588864 | 95660 / 226680 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9198784824 | 6437847040 | 4110794752 | 2603536384 / 7511588864 | 94704 / 253010 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.8948369973 | 6437847040 | 3740631040 | 3179642880 / 9530753024 | 96785 / 286707 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8946682048 | 6437847040 | 4256780288 | 2567884800 / 7511588864 | 95533 / 242609 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:01:33,538 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:03:00,661 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210910616 | 324906015 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319777712 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 814533768 | 627370453 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:03:00,663 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:03:11,013 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 01:03:12,667 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 01:03:12,668 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_rebalance_out 2021-05-17 01:03:12,670 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:03:23,164 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:03:23,381 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 01:03:23,381 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:03:37,448 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2409 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2467 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2434 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2405 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2416 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2462 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2431 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2422 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2400 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:37,450 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2409 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2434 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2467 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2405 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2416 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2462 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2431 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-2422 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2400 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:38,805 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 9 docs from 0 to 5000 2021-05-17 01:03:41,943 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4181 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4151 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4173 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4160 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4166 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4123 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4175 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4142 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4011 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4158 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4169 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:41,946 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4192 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4181 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4184 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4151 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4173 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4160 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4129 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4166 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4123 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4142 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4175 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4011 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS] | | test_collections-4158 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4169 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4136 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,585 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4735 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4758 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4712 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4723 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4717 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4726 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4682 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4770 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4687 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4775 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4797 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4798 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4666 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4763 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4730 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,588 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4708 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4757 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4735 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4758 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4723 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4712 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4717 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4739 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4726 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4682 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4770 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4687 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4698 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4797 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4775 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4666 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4798 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4730 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4763 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,625 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 19 docs from 0 to 5000 2021-05-17 01:03:42,628 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 15 docs from 0 to 5000 2021-05-17 01:03:44,098 | test | INFO | pool-5-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5722.713 | | 1468.823 | ........................X | 1468.823 | | 59.945 | X | 59.945 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:03:44,141 | test | INFO | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 2626.305 | | 1917.360 | ........................................................................X | 1917.360 | | 586.883 | .....................X | 586.883 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:03:45,553 | test | INFO | pool-5-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......................................................................................*.............X | 21005.938 | | 20847.741 | ..................................................................................................X | 20847.741 | | 20779.914 | .................................................................................................X | 20779.914 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:03:57,592 | test | INFO | pool-5-thread-9 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.20700001717 seconds to complete 2021-05-17 01:03:57,599 | test | INFO | pool-5-thread-9 | [common_lib:sleep:22] Sleep 8.20700001717 seconds. Reason: Wait after rebalance complete 2021-05-17 01:06:05,869 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0589997768402 seconds 2021-05-17 01:06:17,476 | test | INFO | pool-5-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.48200011253 seconds to complete 2021-05-17 01:06:17,482 | test | INFO | pool-5-thread-6 | [common_lib:sleep:22] Sleep 8.48200011253 seconds. Reason: Wait after rebalance complete 2021-05-17 01:08:26,022 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 01:08:26,023 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:08:26,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:08:33,029 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:08:36,440 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,512 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,622 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,684 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,954 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,954 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,029 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,115 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,216 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,407 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,459 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:38,464 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:38,700 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,760 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,877 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,969 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,134 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,263 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,493 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,572 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,002 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,282 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,426 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,434 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,489 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,497 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,578 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,663 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,697 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,773 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,512 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:42,516 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 111 docs from 1000 to 1960 2021-05-17 01:08:42,642 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,730 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,835 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,888 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,993 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:43,022 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:43,147 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:44,976 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:45,023 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:45,086 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:46,233 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,243 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,253 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,260 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:49,492 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..............*......................................................................................X | 3960.129 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:08:49,572 | test | INFO | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*...........................................................................................X | 4115.282 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:08:49,740 | test | INFO | pool-5-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................*...............................................................................X | 15759.419 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:08:49,832 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,834 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,894 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,894 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:09:07,670 | test | INFO | pool-5-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.726690889696 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 73.8 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 85 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 62.6041736116 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.73 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:09:07,693 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:09:12,776 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.41 2021-05-17 01:09:17,887 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 8.99 2021-05-17 01:09:22,973 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 17.09 2021-05-17 01:09:28,058 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 27.51 2021-05-17 01:09:33,365 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 01:09:38,446 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 44.99 2021-05-17 01:09:43,601 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.59 2021-05-17 01:09:48,673 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:09:53,737 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.29 2021-05-17 01:09:58,826 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 80.84 2021-05-17 01:10:03,900 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 88.53 2021-05-17 01:10:08,967 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 95.46 2021-05-17 01:10:14,094 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 01:10:24,184 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:10:24,342 | test | INFO | pool-5-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 76.6729998589 sec 2021-05-17 01:10:24,344 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 01:10:26,309 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 01:10:27,276 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:10:27,276 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:10:39,993 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:10:39,993 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:10:55,023 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:10:55,025 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:11:51,622 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.135999918 seconds to complete 2021-05-17 01:11:51,638 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 979.373s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:12:01,002 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:12:01,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:12:02,122 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dbe type=client open_count=1 channel=[id: 0x44871791, 0.0.0.0/0.0.0.0:48925] timeout=300.0>) 2021-05-17 01:12:22,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dbf type=client open_count=1 channel=[id: 0x74a7c330, 0.0.0.0/0.0.0.0:51529] timeout=300.0>) 2021-05-17 01:12:24,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dc0 type=client open_count=1 channel=[id: 0xf1a2d509, 0.0.0.0/0.0.0.0:57167] timeout=300.0>) 2021-05-17 01:12:26,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc1 type=client open_count=1 channel=[id: 0x7264de9a, 0.0.0.0/0.0.0.0:43414] timeout=300.0>) 2021-05-17 01:12:51,380 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc2 type=client open_count=1 channel=[id: 0x0ad626fa, 0.0.0.0/0.0.0.0:35832] timeout=300.0>) 2021-05-17 01:12:53,404 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc3 type=client open_count=1 channel=[id: 0xb30e846f, 0.0.0.0/0.0.0.0:50787] timeout=300.0>) 2021-05-17 01:12:55,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc4 type=client open_count=1 channel=[id: 0xbfa8bd60, 0.0.0.0/0.0.0.0:45145] timeout=300.0>) 2021-05-17 01:13:19,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc5 type=client open_count=1 channel=[id: 0xb5f72826, 0.0.0.0/0.0.0.0:37941] timeout=300.0>) 2021-05-17 01:13:21,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc6 type=client open_count=1 channel=[id: 0x9b840ffc, 0.0.0.0/0.0.0.0:41204] timeout=300.0>) 2021-05-17 01:13:23,835 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc7 type=client open_count=1 channel=[id: 0x311f6ce4, 0.0.0.0/0.0.0.0:59078] timeout=300.0>) 2021-05-17 01:13:49,127 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc8 type=client open_count=1 channel=[id: 0x819cb21d, 0.0.0.0/0.0.0.0:60464] timeout=300.0>) 2021-05-17 01:13:51,145 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc9 type=client open_count=1 channel=[id: 0x386f1fc1, 0.0.0.0/0.0.0.0:47323] timeout=300.0>) 2021-05-17 01:13:53,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dca type=client open_count=1 channel=[id: 0x8041e755, 0.0.0.0/0.0.0.0:50512] timeout=300.0>) 2021-05-17 01:13:55,181 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dcb type=client open_count=1 channel=[id: 0x8f2b126a, 0.0.0.0/0.0.0.0:46574] timeout=300.0>) 2021-05-17 01:13:57,200 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dcc type=client open_count=1 channel=[id: 0x3a291f1f, 0.0.0.0/0.0.0.0:53683] timeout=300.0>) 2021-05-17 01:13:59,217 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dcd type=client open_count=1 channel=[id: 0xf0058cf4, 0.0.0.0/0.0.0.0:40129] timeout=300.0>) 2021-05-17 01:14:23,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dce type=client open_count=1 channel=[id: 0xfec4ed39, 0.0.0.0/0.0.0.0:60225] timeout=300.0>) 2021-05-17 01:14:25,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dcf type=client open_count=1 channel=[id: 0x25e895d6, 0.0.0.0/0.0.0.0:59273] timeout=300.0>) 2021-05-17 01:14:27,525 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:14:44,849 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:14:58,641 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:14:58,664 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:14:58,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:15:12,809 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.00316656111 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:15:12,835 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:15:22,905 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:15:22,931 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1210000515 sec 2021-05-17 01:15:23,183 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:16:11,423 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:16:11,854 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 825103016 | 138629819 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 292133600 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1210977192 | 61923801 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 01:16:11,855 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:16:54,559 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:16:54,571 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:17:16,653 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 197.169 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:17:18,507 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..................................................................................**..................X | 159.482 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:17:19,947 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.**...................................................................................................*X | 24184.038 | | 24162.719 | ..................................................................................................X | 24162.719 | | 23925.579 | .................................................................................................X | 23925.579 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:17:20,214 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.2454845925 | 6437847040 | 4030156800 | 2746650624 / 7511588864 | 51020 / 150232 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.6373048453 | 6437847040 | 4196646912 | 2590498816 / 7511588864 | 93610 / 284097 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9466693332 | 6437847040 | 4194025472 | 2584666112 / 7511588864 | 100400 / 240179 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.3244030285 | 6437847040 | 3812818944 | 3177824256 / 9530753024 | 99128 / 223527 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.6377955372 | 6437847040 | 4154380288 | 2567847936 / 7511588864 | 98305 / 266741 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:17:34,286 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:19:01,371 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211369736 | 314354858 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320191280 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 828010104 | 609052213 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:19:01,371 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:19:11,596 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 01:19:11,598 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 01:19:11,598 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:19:22,786 | test | INFO | pool-6-thread-19 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.23000001907 seconds to complete 2021-05-17 01:19:22,790 | test | INFO | pool-6-thread-19 | [common_lib:sleep:22] Sleep 8.23000001907 seconds. Reason: Wait after rebalance complete 2021-05-17 01:21:31,082 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0579998493195 seconds 2021-05-17 01:21:42,753 | test | INFO | pool-6-thread-5 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.16799998283 seconds to complete 2021-05-17 01:21:42,757 | test | INFO | pool-6-thread-5 | [common_lib:sleep:22] Sleep 8.16799998283 seconds. Reason: Wait after rebalance complete 2021-05-17 01:23:50,986 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0570001602173 seconds 2021-05-17 01:23:50,986 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:23:50,987 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:23:58,273 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:24:13,986 | test | INFO | pool-6-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..............................................*......................................................X | 1161.150 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:24:14,822 | test | INFO | pool-6-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 1604.391 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:24:14,861 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................*X | 13291.347 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:24:14,974 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:24:14,976 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:24:15,042 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:24:15,042 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:24:33,118 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:24:33,346 | test | INFO | pool-6-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.521657972367 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 72.7189263179 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 70.8749226731 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 64.765 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.548324194597 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:24:33,398 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:24:38,625 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:24:40,936 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:24:43,845 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.92 2021-05-17 01:24:49,157 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.92 2021-05-17 01:24:55,318 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 2.78 2021-05-17 01:25:00,247 | test | INFO | pool-6-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *........................................................*............................................*X | 2088.676 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:25:00,523 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 3.76 2021-05-17 01:25:01,934 | test | INFO | pool-6-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*........**...................................................................................X | 3378.401 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:25:02,246 | test | INFO | pool-6-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................................X | 5642.696 | | 5642.696 | ....................................*...............................................................*X | 15099.033 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:25:05,621 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 6.53 2021-05-17 01:25:10,700 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 9.36 2021-05-17 01:25:15,798 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.61 2021-05-17 01:25:20,914 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.93 2021-05-17 01:25:26,065 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.11 2021-05-17 01:25:31,151 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.39 2021-05-17 01:25:36,232 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.34 2021-05-17 01:25:41,322 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 44.26 2021-05-17 01:25:46,433 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 51.73 2021-05-17 01:25:51,529 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 59.88 2021-05-17 01:25:56,619 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:26:01,687 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 71.94 2021-05-17 01:26:06,779 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 78.08 2021-05-17 01:26:11,859 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 84.53 2021-05-17 01:26:16,941 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 91.04 2021-05-17 01:26:22,023 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 97.43 2021-05-17 01:26:32,137 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:26:32,286 | test | INFO | pool-6-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 118.940999985 sec 2021-05-17 01:26:33,292 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:26:33,292 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:26:46,046 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:26:46,048 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:27:05,269 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:27:05,269 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:28:00,411 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-17 01:28:00,427 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 968.763s During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 5, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:28:09,954 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:28:10,617 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:28:11,043 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f0 type=client open_count=1 channel=[id: 0x07d297e3, 0.0.0.0/0.0.0.0:60180] timeout=300.0>) 2021-05-17 01:28:31,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f1 type=client open_count=1 channel=[id: 0xc8ac43f2, 0.0.0.0/0.0.0.0:59511] timeout=300.0>) 2021-05-17 01:28:33,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f2 type=client open_count=1 channel=[id: 0x90162b89, 0.0.0.0/0.0.0.0:55137] timeout=300.0>) 2021-05-17 01:28:35,897 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f3 type=client open_count=1 channel=[id: 0x2afb44ac, 0.0.0.0/0.0.0.0:41323] timeout=300.0>) 2021-05-17 01:28:59,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f4 type=client open_count=1 channel=[id: 0x6e49327e, 0.0.0.0/0.0.0.0:60719] timeout=300.0>) 2021-05-17 01:29:01,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f5 type=client open_count=1 channel=[id: 0x9ad59d96, 0.0.0.0/0.0.0.0:42588] timeout=300.0>) 2021-05-17 01:29:03,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f6 type=client open_count=1 channel=[id: 0x56d3defa, 0.0.0.0/0.0.0.0:33068] timeout=300.0>) 2021-05-17 01:29:27,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f7 type=client open_count=1 channel=[id: 0x234f468f, 0.0.0.0/0.0.0.0:55615] timeout=300.0>) 2021-05-17 01:29:29,862 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f8 type=client open_count=1 channel=[id: 0x6f822940, 0.0.0.0/0.0.0.0:48443] timeout=300.0>) 2021-05-17 01:29:31,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7f9 type=client open_count=1 channel=[id: 0x694fa3c8, 0.0.0.0/0.0.0.0:57900] timeout=300.0>) 2021-05-17 01:29:56,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7fa type=client open_count=1 channel=[id: 0x62443c46, 0.0.0.0/0.0.0.0:50138] timeout=300.0>) 2021-05-17 01:29:58,599 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7fb type=client open_count=1 channel=[id: 0x48af4212, 0.0.0.0/0.0.0.0:41224] timeout=300.0>) 2021-05-17 01:30:00,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fc type=client open_count=1 channel=[id: 0xa47f004d, 0.0.0.0/0.0.0.0:59331] timeout=300.0>) 2021-05-17 01:30:24,759 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fd type=client open_count=1 channel=[id: 0xf1891a05, 0.0.0.0/0.0.0.0:40322] timeout=300.0>) 2021-05-17 01:30:26,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fe type=client open_count=1 channel=[id: 0xe6c84fd4, 0.0.0.0/0.0.0.0:37271] timeout=300.0>) 2021-05-17 01:30:28,795 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:30:46,052 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:30:59,707 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:30:59,729 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:30:59,766 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:31:13,875 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.66158897352 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:31:13,901 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:31:23,980 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:31:24,007 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1339998245 sec 2021-05-17 01:31:24,273 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:32:16,796 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:32:17,151 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 876682040 | 135632571 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 297069504 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 929359320 | 107314098 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:32:17,151 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:33:01,072 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:33:01,085 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:33:23,234 | test | INFO | pool-7-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.................................**...................................................................*X | 24735.895 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:33:24,733 | test | INFO | pool-7-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....***...............................................................................................*X | 146.677 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:33:25,388 | test | INFO | pool-7-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...........................**.........................................................................**X | 167.644 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:33:25,654 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.6606202458 | 6437847040 | 4158156800 | 2617679872 / 7511588864 | 67670 / 198927 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 98.4383073051 | 6437847040 | 4208533504 | 2569297920 / 7511588864 | 67116 / 81341 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.7404066962 | 6437847040 | 4152954880 | 2610405376 / 7511588864 | 66092 / 170574 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.973326665 | 6437847040 | 3609845760 | 3354857472 / 9530753024 | 101800 / 287891 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8452424535 | 6437847040 | 4181737472 | 2615734272 / 7511588864 | 66794 / 168941 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:33:40,891 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:35:08,010 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210825048 | 318226215 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320723760 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 833004360 | 613302031 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:35:08,012 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:35:18,207 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 01:35:19,835 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 01:35:19,835 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_recovery 2021-05-17 01:35:19,836 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:35:29,049 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:35:29,226 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 01:35:29,226 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:35:44,930 | test | INFO | pool-7-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...................................................................................................X | 11509.229 | +-------+-------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:35:47,904 | test | INFO | pool-7-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5006.430 | | 106.522 | .X | 106.522 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:35:48,321 | test | INFO | pool-7-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5250.357 | | 530.773 | .........X | 530.773 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:36:56,190 | test | INFO | pool-7-thread-14 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 86.6129999161 seconds to complete 2021-05-17 01:36:56,197 | test | INFO | pool-7-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:39:01,257 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 01:39:46,903 | test | INFO | pool-7-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 45.4580001831 seconds to complete 2021-05-17 01:39:46,907 | test | INFO | pool-7-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:41:51,967 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 01:41:52,161 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:41:52,164 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:41:59,158 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:42:12,201 | test | INFO | pool-7-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 18262.328 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:42:12,394 | test | INFO | pool-7-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 8652.624 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:42:12,482 | test | INFO | pool-7-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5789.830 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:42:12,680 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,680 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,732 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,732 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:42:29,418 | test | INFO | pool-7-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.416680556019 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 81.0947761443 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 66.98110063 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 72.8124270809 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.679977334089 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:42:29,440 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:34,497 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:39,563 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:44,746 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.31 2021-05-17 01:42:49,930 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.79 2021-05-17 01:42:55,204 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 14.52 2021-05-17 01:43:00,500 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 19.94 2021-05-17 01:43:06,013 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 22.89 2021-05-17 01:43:11,463 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 28.1 2021-05-17 01:43:16,742 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 01:43:23,822 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.29 2021-05-17 01:43:29,588 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 53.13 2021-05-17 01:43:34,723 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 58.4 2021-05-17 01:43:40,063 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:43:45,171 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.81 2021-05-17 01:43:50,967 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 75.28 2021-05-17 01:43:56,055 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 79.61 2021-05-17 01:44:01,138 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 83.04 2021-05-17 01:44:06,209 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 85.91 2021-05-17 01:44:11,329 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 90.95 2021-05-17 01:44:16,414 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 92.3 2021-05-17 01:44:21,653 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 98.46 2021-05-17 01:44:31,743 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:44:31,783 | test | INFO | pool-7-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 122.365999937 sec 2021-05-17 01:44:44,157 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:45:58,720 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 01:46:00,719 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 01:46:01,674 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:46:01,676 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:46:14,240 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:46:14,240 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:46:35,573 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:46:35,575 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:47:30,551 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 01:47:30,565 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 Ran 1 test in 1170.185s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 6, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:47:40,234 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:47:41,035 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:47:41,457 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e5f type=client open_count=1 channel=[id: 0x8e9bc38e, 0.0.0.0/0.0.0.0:50976] timeout=300.0>) 2021-05-17 01:48:02,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e60 type=client open_count=1 channel=[id: 0x3e3b91f8, 0.0.0.0/0.0.0.0:39510] timeout=300.0>) 2021-05-17 01:48:04,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e61 type=client open_count=1 channel=[id: 0xb2b39248, 0.0.0.0/0.0.0.0:57881] timeout=300.0>) 2021-05-17 01:48:06,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e62 type=client open_count=1 channel=[id: 0x72e81b02, 0.0.0.0/0.0.0.0:59983] timeout=300.0>) 2021-05-17 01:48:30,150 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e63 type=client open_count=1 channel=[id: 0xb46d55e3, 0.0.0.0/0.0.0.0:34739] timeout=300.0>) 2021-05-17 01:48:32,168 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e64 type=client open_count=1 channel=[id: 0x68b8f4a9, 0.0.0.0/0.0.0.0:57525] timeout=300.0>) 2021-05-17 01:48:34,186 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e65 type=client open_count=1 channel=[id: 0x456eb8d6, 0.0.0.0/0.0.0.0:58685] timeout=300.0>) 2021-05-17 01:48:58,158 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e66 type=client open_count=1 channel=[id: 0x2beb4074, 0.0.0.0/0.0.0.0:34432] timeout=300.0>) 2021-05-17 01:49:00,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e67 type=client open_count=1 channel=[id: 0xff5e2815, 0.0.0.0/0.0.0.0:50019] timeout=300.0>) 2021-05-17 01:49:02,194 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e68 type=client open_count=1 channel=[id: 0x0ab5470c, 0.0.0.0/0.0.0.0:35960] timeout=300.0>) 2021-05-17 01:49:26,186 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e69 type=client open_count=1 channel=[id: 0x2c8c1613, 0.0.0.0/0.0.0.0:60328] timeout=300.0>) 2021-05-17 01:49:28,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e6a type=client open_count=1 channel=[id: 0x0cc5d59d, 0.0.0.0/0.0.0.0:42156] timeout=300.0>) 2021-05-17 01:49:30,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6b type=client open_count=1 channel=[id: 0x5c73f1e5, 0.0.0.0/0.0.0.0:49283] timeout=300.0>) 2021-05-17 01:49:54,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6c type=client open_count=1 channel=[id: 0x40f4876a, 0.0.0.0/0.0.0.0:34894] timeout=300.0>) 2021-05-17 01:49:56,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6d type=client open_count=1 channel=[id: 0xf0c30c92, 0.0.0.0/0.0.0.0:43172] timeout=300.0>) 2021-05-17 01:49:58,288 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:50:15,482 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:50:29,153 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:50:29,176 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:50:29,213 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:50:43,017 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 11.5898068366 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:50:43,042 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:50:53,115 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:50:53,141 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.125 sec 2021-05-17 01:50:53,394 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:51:46,911 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:51:47,207 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 780461256 | 150646608 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 287649472 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 958313032 | 93412038 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:51:47,207 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:52:29,881 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:52:29,894 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:52:53,505 | test | INFO | pool-8-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......................................*...*............................*..............................X | 167.652 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:52:53,561 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..................**.......................................................*...........................X | 176.450 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:52:54,325 | test | INFO | pool-8-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......................**..............................................................................*X | 26582.246 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:52:54,614 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 100 | 6437847040 | 4136271872 | 2635771904 / 7511588864 | 66721 / 199077 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 81.4132385193 | 6437847040 | 4243619840 | 2539024384 / 7511588864 | 64700 / 150690 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 84.5816441816 | 6437847040 | 4263325696 | 2506563584 / 7511588864 | 66317 / 197277 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 83.7487935568 | 6437847040 | 3862028288 | 3106611200 / 9530753024 | 66597 / 175073 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 82.3222017618 | 6437847040 | 4297330688 | 2500845568 / 7511588864 | 66070 / 164061 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:53:09,421 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:54:36,729 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211783656 | 312935045 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319237296 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 842485320 | 619182210 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:54:36,730 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:55:46,822 | test | INFO | pool-8-thread-16 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.7079999447 seconds to complete 2021-05-17 01:55:46,828 | test | INFO | pool-8-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:57:51,888 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 01:58:34,700 | test | INFO | pool-8-thread-8 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 42.631000042 seconds to complete 2021-05-17 01:58:34,710 | test | INFO | pool-8-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:00:39,828 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.115000009537 seconds 2021-05-17 02:00:39,828 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:00:39,829 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:00:47,345 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:01:05,858 | test | INFO | pool-8-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 2231.392 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:06,885 | test | INFO | pool-8-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***...................................................................................................**X | 3157.434 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:07,720 | test | INFO | pool-8-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................*.......................................**...........................................X | 20446.615 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:01:08,019 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,020 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,148 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,150 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:01:22,829 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:01:23,105 | test | INFO | pool-8-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.546666666667 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 84.2980716345 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 83.1499874802 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 59.2716666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.576657055716 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:01:23,267 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:28,510 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:32,266 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:01:33,677 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:38,905 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:44,142 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:50,470 | test | INFO | pool-8-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................X | 5134.506 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:51,552 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:52,575 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........***..........................................................................................*X | 4447.726 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:53,000 | test | INFO | pool-8-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........*......**...................................................................................*X | 18994.644 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:01:56,690 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 4.23 2021-05-17 02:02:01,888 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 7.29 2021-05-17 02:02:07,056 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 10.67 2021-05-17 02:02:12,229 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 13.98 2021-05-17 02:02:17,359 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 17.69 2021-05-17 02:02:22,499 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.73 2021-05-17 02:02:27,591 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 25.6 2021-05-17 02:02:32,757 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 29.12 2021-05-17 02:02:37,865 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 32.13 2021-05-17 02:02:44,581 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 34.32 2021-05-17 02:02:49,780 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 43.81 2021-05-17 02:02:55,055 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 50.71 2021-05-17 02:03:00,480 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 57.54 2021-05-17 02:03:05,789 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 63.23 2021-05-17 02:03:10,950 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 02:03:16,040 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.16 2021-05-17 02:03:21,240 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 75.64 2021-05-17 02:03:26,433 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.87 2021-05-17 02:03:31,585 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 83.51 2021-05-17 02:03:36,770 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 86.35 2021-05-17 02:03:42,009 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 90.13 2021-05-17 02:03:47,247 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 93.55 2021-05-17 02:03:52,451 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 96.17 2021-05-17 02:03:57,605 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 98.75 2021-05-17 02:04:07,838 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:04:07,934 | test | INFO | pool-8-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 164.828999996 sec 2021-05-17 02:04:34,028 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:06:23,842 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:06:23,844 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:06:42,855 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 02:06:42,855 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 02:07:06,572 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:07:06,572 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:08:12,371 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.2690000534 seconds to complete 2021-05-17 02:08:12,387 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1242.445s OK During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 7, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:08:27,801 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:08:29,414 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:08:31,522 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a744 type=client open_count=1 channel=[id: 0x29521add, 0.0.0.0/0.0.0.0:45172] timeout=300.0>) 2021-05-17 02:08:56,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a745 type=client open_count=1 channel=[id: 0x5c81ece5, 0.0.0.0/0.0.0.0:40790] timeout=300.0>) 2021-05-17 02:08:58,821 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a746 type=client open_count=1 channel=[id: 0x01dd2be8, 0.0.0.0/0.0.0.0:56805] timeout=300.0>) 2021-05-17 02:09:00,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a747 type=client open_count=1 channel=[id: 0xf61703d9, 0.0.0.0/0.0.0.0:34973] timeout=300.0>) 2021-05-17 02:09:26,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a748 type=client open_count=1 channel=[id: 0xd5a93556, 0.0.0.0/0.0.0.0:52391] timeout=300.0>) 2021-05-17 02:09:28,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a749 type=client open_count=1 channel=[id: 0xc019a27b, 0.0.0.0/0.0.0.0:50454] timeout=300.0>) 2021-05-17 02:09:30,479 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74a type=client open_count=1 channel=[id: 0x8ca75604, 0.0.0.0/0.0.0.0:39009] timeout=300.0>) 2021-05-17 02:09:55,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74b type=client open_count=1 channel=[id: 0xa3a00154, 0.0.0.0/0.0.0.0:46030] timeout=300.0>) 2021-05-17 02:09:57,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74c type=client open_count=1 channel=[id: 0x04180ad8, 0.0.0.0/0.0.0.0:52609] timeout=300.0>) 2021-05-17 02:09:59,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74d type=client open_count=1 channel=[id: 0x5de30d20, 0.0.0.0/0.0.0.0:55174] timeout=300.0>) 2021-05-17 02:10:24,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74e type=client open_count=1 channel=[id: 0xee7a1d07, 0.0.0.0/0.0.0.0:58226] timeout=300.0>) 2021-05-17 02:10:26,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74f type=client open_count=1 channel=[id: 0x6750bbd0, 0.0.0.0/0.0.0.0:39185] timeout=300.0>) 2021-05-17 02:10:28,783 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a750 type=client open_count=1 channel=[id: 0x4614972f, 0.0.0.0/0.0.0.0:45999] timeout=300.0>) 2021-05-17 02:10:53,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a751 type=client open_count=1 channel=[id: 0xed6ab0d8, 0.0.0.0/0.0.0.0:59190] timeout=300.0>) 2021-05-17 02:10:55,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a752 type=client open_count=1 channel=[id: 0x88b36274, 0.0.0.0/0.0.0.0:42896] timeout=300.0>) 2021-05-17 02:10:57,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:11:17,526 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:11:31,512 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:11:31,536 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:11:31,573 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:11:45,163 | test | INFO | pool-9-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.68507808463 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 02:11:45,187 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:11:55,259 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:11:55,285 | test | INFO | pool-9-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.123000145 sec 2021-05-17 02:11:55,535 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:12:41,890 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:12:42,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 925929048 | 108210094 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 291696080 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1295415608 | 43904602 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 02:12:42,232 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:13:25,657 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:13:25,667 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:13:46,934 | test | INFO | pool-9-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 22781.131 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:13:47,032 | test | INFO | pool-9-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 157.415 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:13:47,088 | test | INFO | pool-9-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 149.829 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:13:47,357 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.5573822321 | 6437847040 | 4200837120 | 2583797760 / 7511588864 | 101251 / 245391 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.7689736532 | 6437847040 | 4204818432 | 2580123648 / 7511588864 | 100083 / 237747 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 98.7027851965 | 6437847040 | 4169306112 | 2603659264 / 7511588864 | 44688 / 115740 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.950154521 | 6437847040 | 3749453824 | 3241103360 / 9530753024 | 101065 / 266063 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.4823982216 | 6437847040 | 4187570176 | 2610003968 / 7511588864 | 47850 / 139215 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:14:02,081 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:15:28,723 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212902360 | 297653854 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320312800 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 859893512 | 604754918 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:15:28,724 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:15:42,244 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 02:15:46,467 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 02:15:46,467 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_recovery 2021-05-17 02:15:46,469 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:15:56,516 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:15:56,688 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 02:15:56,690 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:16:15,446 | test | INFO | pool-9-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......................................*.............................................................X | 16002.363 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:16:17,319 | test | INFO | pool-9-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 3261.444 | | 129.262 | ..X | 129.262 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:16:17,549 | test | INFO | pool-9-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..*.................................................................................................X | 1730.482 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:17:13,114 | test | INFO | pool-9-thread-9 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 75.6370000839 seconds to complete 2021-05-17 02:17:13,121 | test | INFO | pool-9-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:19:18,186 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0610001087189 seconds 2021-05-17 02:19:55,132 | test | INFO | pool-9-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 36.7649998665 seconds to complete 2021-05-17 02:19:55,141 | test | INFO | pool-9-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:22:00,203 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0590000152588 seconds 2021-05-17 02:22:00,355 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:22:00,357 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:22:07,323 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:22:18,542 | test | INFO | pool-9-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5248.537 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:19,367 | test | INFO | pool-9-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 4548.921 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:20,954 | test | INFO | pool-9-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................................................................................*..................X | 5264.472 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:21,163 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,164 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,221 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,223 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:22:37,283 | test | INFO | pool-9-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.516701113408 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 79.2666666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 73.5736833361 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 75.8382774482 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.570019000633 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:22:37,308 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:42,368 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:47,421 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:53,615 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:58,811 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.95 2021-05-17 02:23:03,923 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 6.08 2021-05-17 02:23:09,203 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 10.22 2021-05-17 02:23:14,414 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 17.02 2021-05-17 02:23:19,528 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 22.04 2021-05-17 02:23:24,638 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 02:23:29,726 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.88 2021-05-17 02:23:36,828 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 35.21 2021-05-17 02:23:41,926 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 42.66 2021-05-17 02:23:47,078 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 46.59 2021-05-17 02:23:52,351 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 49.18 2021-05-17 02:23:57,914 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 53.33 2021-05-17 02:24:04,737 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 62.25 2021-05-17 02:24:10,038 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 72.08 2021-05-17 02:24:15,198 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-17 02:24:25,302 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:24:25,341 | test | INFO | pool-9-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 108.055999994 sec 2021-05-17 02:24:37,793 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:25:53,664 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 02:25:55,642 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 02:25:56,619 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:25:56,621 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:26:13,388 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 02:26:13,388 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 02:26:29,523 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:26:29,523 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:27:20,776 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-17 02:27:20,790 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1147.737s OK During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 8, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:27:30,144 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:27:30,836 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:27:31,256 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f24e type=client open_count=1 channel=[id: 0x19271d0a, 0.0.0.0/0.0.0.0:45923] timeout=300.0>) 2021-05-17 02:27:52,832 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f24f type=client open_count=1 channel=[id: 0xbf4f992a, 0.0.0.0/0.0.0.0:37739] timeout=300.0>) 2021-05-17 02:27:54,872 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f250 type=client open_count=1 channel=[id: 0x34142a6d, 0.0.0.0/0.0.0.0:33298] timeout=300.0>) 2021-05-17 02:27:56,918 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f251 type=client open_count=1 channel=[id: 0x92e46cbd, 0.0.0.0/0.0.0.0:52782] timeout=300.0>) 2021-05-17 02:28:23,749 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f252 type=client open_count=1 channel=[id: 0x89e7f94b, 0.0.0.0/0.0.0.0:41692] timeout=300.0>) 2021-05-17 02:28:25,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f253 type=client open_count=1 channel=[id: 0x97188be9, 0.0.0.0/0.0.0.0:55414] timeout=300.0>) 2021-05-17 02:28:27,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f254 type=client open_count=1 channel=[id: 0x75174782, 0.0.0.0/0.0.0.0:47245] timeout=300.0>) 2021-05-17 02:28:54,263 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f255 type=client open_count=1 channel=[id: 0x9cfe9fc7, 0.0.0.0/0.0.0.0:60196] timeout=300.0>) 2021-05-17 02:28:56,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f256 type=client open_count=1 channel=[id: 0x14dcd05f, 0.0.0.0/0.0.0.0:54912] timeout=300.0>) 2021-05-17 02:28:58,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f257 type=client open_count=1 channel=[id: 0x5e396793, 0.0.0.0/0.0.0.0:54428] timeout=300.0>) 2021-05-17 02:29:25,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f258 type=client open_count=1 channel=[id: 0xa32727fe, 0.0.0.0/0.0.0.0:49706] timeout=300.0>) 2021-05-17 02:29:27,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f259 type=client open_count=1 channel=[id: 0xe05d7c9a, 0.0.0.0/0.0.0.0:43710] timeout=300.0>) 2021-05-17 02:29:29,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25a type=client open_count=1 channel=[id: 0x407b8769, 0.0.0.0/0.0.0.0:51359] timeout=300.0>) 2021-05-17 02:29:55,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25b type=client open_count=1 channel=[id: 0xdb06c792, 0.0.0.0/0.0.0.0:48522] timeout=300.0>) 2021-05-17 02:29:57,581 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25c type=client open_count=1 channel=[id: 0x6fd5d9a1, 0.0.0.0/0.0.0.0:40022] timeout=300.0>) 2021-05-17 02:29:59,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:30:19,052 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:30:34,578 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:30:34,601 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:30:34,638 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:30:48,232 | test | INFO | pool-10-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.14020701035 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 02:30:48,272 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:30:58,404 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:30:58,457 | test | INFO | pool-10-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2239999771 sec 2021-05-17 02:30:59,020 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:31:54,210 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:31:54,990 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 805387936 | 153067684 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 283598864 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 720892248 | 122986010 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 02:31:54,990 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:32:50,368 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:32:50,377 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:33:15,266 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 165.054 | | 74.319 | ............................................*X | 74.319 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:33:16,631 | test | INFO | pool-10-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................................***.................................................................**X | 21515.831 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:33:17,434 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 156.557 | | 62.324 | ......................................**....X | 67.636 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:33:17,734 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.2944621938 | 6437847040 | 4182282240 | 2588782592 / 7511588864 | 59333 / 179322 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.4284574998 | 6437847040 | 4161486848 | 2610003968 / 7511588864 | 58727 / 157168 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.2744458497 | 6437847040 | 4178894848 | 2587127808 / 7511588864 | 100400 / 303800 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.4800086665 | 6437847040 | 3746578432 | 3203096576 / 9530753024 | 58960 / 146429 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 98.3586301324 | 6437847040 | 4191698944 | 2611060736 / 7511588864 | 101600 / 301000 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:33:36,380 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:35:27,805 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211093320 | 311551038 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319196752 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 811598376 | 644304730 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:35:27,808 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:35:43,335 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 02:35:43,336 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 02:35:43,338 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:36:42,918 | test | INFO | pool-10-thread-25 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.2319998741 seconds to complete 2021-05-17 02:36:42,924 | test | INFO | pool-10-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:38:48,036 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.108999967575 seconds 2021-05-17 02:39:31,638 | test | INFO | pool-10-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 43.3489999771 seconds to complete 2021-05-17 02:39:31,645 | test | INFO | pool-10-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:41:36,704 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 02:41:36,704 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:41:36,706 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:41:43,438 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:42:00,408 | test | INFO | pool-10-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 3090.114 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:01,220 | test | INFO | pool-10-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................X | 23522.986 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:42:02,404 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 3953.639 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:02,653 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,653 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,710 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,711 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:42:19,941 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:42:20,191 | test | INFO | pool-10-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.549972501375 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 74.4649951591 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 64.9466666667 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 80.2346705778 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.418326361227 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:42:20,260 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:25,348 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:26,877 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:42:31,230 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:36,454 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:41,559 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:41,572 | test | INFO | pool-10-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 4142.002 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:42,198 | test | INFO | pool-10-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 13150.754 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:42:44,584 | test | INFO | pool-10-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 5335.378 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:46,663 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:51,946 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:57,023 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:43:02,095 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.46 2021-05-17 02:43:07,263 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 1.71 2021-05-17 02:43:12,523 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 6.32 2021-05-17 02:43:17,704 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 11.39 2021-05-17 02:43:22,829 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 15.79 2021-05-17 02:43:27,970 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 19.43 2021-05-17 02:43:33,131 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 24.17 2021-05-17 02:43:39,885 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 02:43:45,141 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 30.76 2021-05-17 02:43:50,326 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 36.88 2021-05-17 02:43:55,489 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 40.28 2021-05-17 02:44:00,766 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 43.74 2021-05-17 02:44:06,019 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 47.61 2021-05-17 02:44:11,233 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 51.48 2021-05-17 02:44:17,007 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 55.35 2021-05-17 02:44:22,217 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 61.94 2021-05-17 02:44:27,443 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 67.77 2021-05-17 02:44:32,601 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 73.72 2021-05-17 02:44:42,782 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:44:42,819 | test | INFO | pool-10-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 142.628999949 sec 2021-05-17 02:44:55,292 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:46:12,086 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:46:12,086 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:46:27,497 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 98% 2021-05-17 02:46:27,499 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 95% 2021-05-17 02:46:38,974 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:46:38,974 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:47:46,826 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 02:47:46,841 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok During the test, ---------------------------------------------------------------------- Ran 1 test in 1226.031s Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 9, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:47:56,348 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:47:57,046 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:47:57,463 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3215 type=client open_count=1 channel=[id: 0x91e65596, 0.0.0.0/0.0.0.0:55411] timeout=300.0>) 2021-05-17 02:48:18,216 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3216 type=client open_count=1 channel=[id: 0x3c8d7ce6, 0.0.0.0/0.0.0.0:53054] timeout=300.0>) 2021-05-17 02:48:20,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3217 type=client open_count=1 channel=[id: 0x310b57cc, 0.0.0.0/0.0.0.0:58618] timeout=300.0>) 2021-05-17 02:48:22,252 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b3218 type=client open_count=1 channel=[id: 0x0dc7c7bc, 0.0.0.0/0.0.0.0:41750] timeout=300.0>) 2021-05-17 02:48:46,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b3219 type=client open_count=1 channel=[id: 0x445c061e, 0.0.0.0/0.0.0.0:57257] timeout=300.0>) 2021-05-17 02:48:48,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b321a type=client open_count=1 channel=[id: 0xef98fa87, 0.0.0.0/0.0.0.0:34609] timeout=300.0>) 2021-05-17 02:48:50,591 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321b type=client open_count=1 channel=[id: 0xc376d9db, 0.0.0.0/0.0.0.0:56538] timeout=300.0>) 2021-05-17 02:49:14,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321c type=client open_count=1 channel=[id: 0xbc7a599e, 0.0.0.0/0.0.0.0:51960] timeout=300.0>) 2021-05-17 02:49:16,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321d type=client open_count=1 channel=[id: 0x69dcde71, 0.0.0.0/0.0.0.0:44541] timeout=300.0>) 2021-05-17 02:49:18,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b321e type=client open_count=1 channel=[id: 0xaf60496a, 0.0.0.0/0.0.0.0:33524] timeout=300.0>) 2021-05-17 02:49:42,884 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b321f type=client open_count=1 channel=[id: 0x8e48e806, 0.0.0.0/0.0.0.0:58203] timeout=300.0>) 2021-05-17 02:49:44,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b3220 type=client open_count=1 channel=[id: 0x8d2b7f95, 0.0.0.0/0.0.0.0:54513] timeout=300.0>) 2021-05-17 02:49:46,920 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3221 type=client open_count=1 channel=[id: 0xfd6f2423, 0.0.0.0/0.0.0.0:49449] timeout=300.0>) 2021-05-17 02:50:11,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3222 type=client open_count=1 channel=[id: 0x25198c38, 0.0.0.0/0.0.0.0:43300] timeout=300.0>) 2021-05-17 02:50:13,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3223 type=client open_count=1 channel=[id: 0xd9046e3b, 0.0.0.0/0.0.0.0:52173] timeout=300.0>) 2021-05-17 02:50:15,091 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:50:32,338 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:50:46,025 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:50:46,046 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:50:46,084 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:51:00,042 | test | INFO | pool-11-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 15.155 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------+--------------+ 2021-05-17 02:51:00,069 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:51:10,142 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:51:10,168 | test | INFO | pool-11-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1259999275 sec 2021-05-17 02:51:10,418 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:52:07,040 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:52:07,415 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 737052152 | 153291653 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 304136576 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 806693864 | 118008106 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 02:52:07,417 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:52:47,184 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:52:47,197 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:53:08,397 | test | INFO | pool-11-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.............................................................................................*.......*X | 240.802 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:53:08,477 | test | INFO | pool-11-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **........................................................................*...........................*X | 221.299 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:53:08,594 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....................................*...............................................................*X | 17316.022 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:53:08,887 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.7642613522 | 6437847040 | 4221870080 | 2556145664 / 7511588864 | 36218 / 107423 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.6594721819 | 6437847040 | 4207136768 | 2555310080 / 7511588864 | 30526 / 80328 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9216679722 | 6437847040 | 4211765248 | 2554540032 / 7511588864 | 39364 / 105093 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.7661032132 | 6437847040 | 3775979520 | 3186417664 / 9530753024 | 35892 / 87598 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.5043204501 | 6437847040 | 4240412672 | 2558140416 / 7511588864 | 35722 / 89544 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:53:23,844 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:54:50,667 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210629208 | 314634507 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319167856 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 846689368 | 630570239 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:54:50,667 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:55:00,891 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 02:55:02,509 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 02:55:02,509 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_recovery 2021-05-17 02:55:02,510 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:55:10,747 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:55:10,920 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 02:55:10,921 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:55:22,260 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket2:idQU6seGSZGDX4bL-7-358000:KZm6BjaG6B12S-7-622000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3271 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3227 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3349 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3209 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3309 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3285 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3262 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3240 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3267 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3222 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3200 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3283 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3216 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3259 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3338 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3350 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3331 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3253 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3256 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3299 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3356 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3213 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:22,263 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket2:idQU6seGSZGDX4bL-7-358000:KZm6BjaG6B12S-7-622000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3290 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3271 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3227 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3205 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3349 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3209 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3309 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3285 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3262 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3240 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3267 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3245 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3300 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3222 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3200 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3345 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3283 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3216 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3259 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3338 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3274 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3350 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3331 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3231 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3253 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3234 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3256 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3299 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3213 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3378 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3356 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:22,733 | test | INFO | pool-11-thread-11 | [table_view:display:72] Keys failed in default:T4-10-507000:aPIoDiKGcZg25-10-507000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:23,832 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in default:n%Zit8F6-9-794000:r6N%AEM8qJhkd-9-994000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4559 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4515 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4438 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4537 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4459 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4596 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4574 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4450 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4593 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4472 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4572 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4550 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4432 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4531 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4477 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4455 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4406 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4403 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4524 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4485 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4464 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4561 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4446 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4468 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4443 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4421 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:23,835 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in default:n%Zit8F6-9-794000:r6N%AEM8qJhkd-9-994000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4438 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4537 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4515 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4559 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4459 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4415 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4437 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4519 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4574 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4596 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4593 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4472 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4450 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4572 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4550 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4410 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4432 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4531 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4455 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4477 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4406 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4403 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4524 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4568 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4528 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4485 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4585 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4464 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4461 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4561 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4489 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4424 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4468 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4446 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4443 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4421 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:24,118 | test | INFO | pool-11-thread-24 | [table_view:display:72] Keys failed in default:scope1:TjvJ8R-10-499000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:24,273 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 37 docs from 0 to 5000 2021-05-17 02:55:25,299 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 31 docs from 0 to 5000 2021-05-17 02:55:25,321 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 37 docs from 0 to 500 2021-05-17 02:55:25,325 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 41 docs from 500 to 980 2021-05-17 02:55:30,200 | test | INFO | pool-11-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......*..............................................................................................X | 4225.989 | | 1553.709 | ...................................X | 1553.709 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:55:31,019 | test | INFO | pool-11-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........................................*..........................................................X | 4299.946 | | 35.806 | *X | 35.806 | +--------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:55:31,961 | test | INFO | pool-11-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................*...............................................................X | 12208.670 | | 9994.626 | ................................................................................*X | 9994.626 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:55:41,756 | test | INFO | pool-11-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.21099996567 seconds to complete 2021-05-17 02:55:41,762 | test | INFO | pool-11-thread-21 | [common_lib:sleep:22] Sleep 8.21099996567 seconds. Reason: Wait after rebalance complete 2021-05-17 02:57:50,030 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-05-17 02:58:01,983 | test | INFO | pool-11-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.31200003624 seconds to complete 2021-05-17 02:58:01,989 | test | INFO | pool-11-thread-24 | [common_lib:sleep:22] Sleep 8.31200003624 seconds. Reason: Wait after rebalance complete 2021-05-17 03:00:10,361 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 03:00:10,361 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:00:10,362 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:00:17,944 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:00:33,278 | test | INFO | pool-11-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........................*...........................................................*...............X | 5895.362 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:00:34,128 | test | INFO | pool-11-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*..........................................................................................X | 7676.942 | | 4187.511 | .....................................................X | 4187.511 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:00:35,598 | test | INFO | pool-11-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..........*...............................................................................**...........X | 22997.150 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:00:35,776 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,778 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,917 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,918 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:00:52,456 | test | INFO | pool-11-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.156661444619 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 62.8633333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 41.7180572686 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 58.4604066346 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.313333333333 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:00:52,519 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:00:57,617 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:02,677 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:07,736 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:13,023 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.41 2021-05-17 03:01:18,391 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 11.08 2021-05-17 03:01:23,575 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.78 2021-05-17 03:01:28,713 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 23.08 2021-05-17 03:01:33,834 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 26.74 2021-05-17 03:01:38,980 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 03:01:45,661 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 39.09 2021-05-17 03:01:50,799 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 49.26 2021-05-17 03:01:55,936 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.71 2021-05-17 03:02:01,039 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 59.86 2021-05-17 03:02:06,121 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.57 2021-05-17 03:02:11,207 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 03:02:16,780 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.63 2021-05-17 03:02:21,926 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 79.01 2021-05-17 03:02:27,055 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 82.77 2021-05-17 03:02:32,196 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 86.22 2021-05-17 03:02:37,309 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 88.31 2021-05-17 03:02:42,405 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 92.5 2021-05-17 03:02:48,354 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 95.32 2021-05-17 03:02:58,470 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:02:58,517 | test | INFO | pool-11-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 126.061999798 sec 2021-05-17 03:02:58,519 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 03:03:01,293 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 03:03:02,628 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:03:02,628 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:03:20,252 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:03:20,253 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:03:39,713 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:03:39,714 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:04:38,119 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1569998264 seconds to complete 2021-05-17 03:04:38,134 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1011.322s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 9 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 10, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:04:48,246 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:04:49,030 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:04:49,575 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548842 type=client open_count=1 channel=[id: 0x8488680e, 0.0.0.0/0.0.0.0:46827] timeout=300.0>) 2021-05-17 03:05:10,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548843 type=client open_count=1 channel=[id: 0xc9c85073, 0.0.0.0/0.0.0.0:52026] timeout=300.0>) 2021-05-17 03:05:12,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548844 type=client open_count=1 channel=[id: 0xdf9a814e, 0.0.0.0/0.0.0.0:43598] timeout=300.0>) 2021-05-17 03:05:14,750 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548845 type=client open_count=1 channel=[id: 0x1e9b8405, 0.0.0.0/0.0.0.0:38336] timeout=300.0>) 2021-05-17 03:05:40,053 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548846 type=client open_count=1 channel=[id: 0xcedd59ed, 0.0.0.0/0.0.0.0:55751] timeout=300.0>) 2021-05-17 03:05:42,104 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548847 type=client open_count=1 channel=[id: 0xd08a7ba5, 0.0.0.0/0.0.0.0:37135] timeout=300.0>) 2021-05-17 03:05:44,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x548848 type=client open_count=1 channel=[id: 0xf999201b, 0.0.0.0/0.0.0.0:50063] timeout=300.0>) 2021-05-17 03:06:08,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x548849 type=client open_count=1 channel=[id: 0x2ca79a91, 0.0.0.0/0.0.0.0:34538] timeout=300.0>) 2021-05-17 03:06:10,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x54884a type=client open_count=1 channel=[id: 0x4cb73f93, 0.0.0.0/0.0.0.0:42994] timeout=300.0>) 2021-05-17 03:06:12,352 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884b type=client open_count=1 channel=[id: 0x302ccf8f, 0.0.0.0/0.0.0.0:53506] timeout=300.0>) 2021-05-17 03:06:36,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884c type=client open_count=1 channel=[id: 0x73a70209, 0.0.0.0/0.0.0.0:38020] timeout=300.0>) 2021-05-17 03:06:38,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884d type=client open_count=1 channel=[id: 0x4b56b36b, 0.0.0.0/0.0.0.0:39241] timeout=300.0>) 2021-05-17 03:06:40,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x54884e type=client open_count=1 channel=[id: 0xf88177e3, 0.0.0.0/0.0.0.0:51214] timeout=300.0>) 2021-05-17 03:07:04,576 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x54884f type=client open_count=1 channel=[id: 0x4ccabc3d, 0.0.0.0/0.0.0.0:52710] timeout=300.0>) 2021-05-17 03:07:06,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x548850 type=client open_count=1 channel=[id: 0xb998f3bd, 0.0.0.0/0.0.0.0:48271] timeout=300.0>) 2021-05-17 03:07:08,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:07:25,951 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:07:39,640 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:07:39,661 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:07:39,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:07:52,983 | test | INFO | pool-12-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-----+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-----+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+-----+--------------+ 2021-05-17 03:07:53,009 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:08:03,082 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:08:03,108 | test | INFO | pool-12-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-17 03:08:03,384 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:08:51,065 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:08:51,414 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 960411712 | 102400167 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 301818688 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1227237224 | 54793741 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 03:08:51,415 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:09:33,733 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:09:33,746 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:09:53,782 | test | INFO | pool-12-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....**................................................................................................X | 27175.967 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:09:54,358 | test | INFO | pool-12-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................X | 148.445 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:09:54,756 | test | INFO | pool-12-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..............................................**......................................................X | 132.650 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:09:55,026 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 71.1891765959 | 6437847040 | 4303298560 | 2456530944 / 7511588864 | 58519 / 136186 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 70.1332623488 | 6437847040 | 4235509760 | 2530041856 / 7511588864 | 57371 / 146457 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 66.7188072021 | 6437847040 | 4247871488 | 2515165184 / 7511588864 | 56769 / 139257 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 68.4213133431 | 6437847040 | 3825758208 | 3108995072 / 9530753024 | 57804 / 169097 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7896739947 | 6437847040 | 4142186496 | 2637008896 / 7511588864 | 56976 / 169147 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:10:10,280 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:11:37,010 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212554840 | 303361380 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319351376 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 824483112 | 608284650 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:11:37,012 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:11:47,178 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 03:11:47,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:11:47,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:11:59,056 | test | INFO | pool-12-thread-3 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.22000002861 seconds to complete 2021-05-17 03:11:59,062 | test | INFO | pool-12-thread-3 | [common_lib:sleep:22] Sleep 8.22000002861 seconds. Reason: Wait after rebalance complete 2021-05-17 03:14:07,345 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 03:14:19,035 | test | INFO | pool-12-thread-20 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.125 seconds to complete 2021-05-17 03:14:19,039 | test | INFO | pool-12-thread-20 | [common_lib:sleep:22] Sleep 8.125 seconds. Reason: Wait after rebalance complete 2021-05-17 03:16:27,224 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 03:16:27,224 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:16:27,226 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:16:35,575 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:16:54,493 | test | INFO | pool-12-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 19648.509 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:16:56,085 | test | INFO | pool-12-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 2885.009 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:16:56,171 | test | INFO | pool-12-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 2170.320 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:16:56,273 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,275 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,332 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,332 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:17:08,144 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:17:08,309 | test | INFO | pool-12-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 1.09331511141 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 89.5075269177 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 75.545 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 85.1386822263 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.393320222659 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:17:08,368 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:13,450 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:16,107 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:17:18,596 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:23,694 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:28,828 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:35,213 | test | INFO | pool-12-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...................................................................................................*X | 14316.942 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:17:35,779 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:37,585 | test | INFO | pool-12-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 3152.126 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:17:37,789 | test | INFO | pool-12-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 4878.498 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:17:40,937 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 4.15 2021-05-17 03:17:46,062 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 6.92 2021-05-17 03:17:51,164 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 9.71 2021-05-17 03:17:56,269 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 12.93 2021-05-17 03:18:01,417 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 16.19 2021-05-17 03:18:06,565 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 20.09 2021-05-17 03:18:11,743 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 24.34 2021-05-17 03:18:16,851 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 27.79 2021-05-17 03:18:21,921 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 30.59 2021-05-17 03:18:27,023 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 03:18:32,119 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 36.96 2021-05-17 03:18:37,272 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 44.98 2021-05-17 03:18:42,404 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 51.18 2021-05-17 03:18:47,496 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 57.17 2021-05-17 03:18:52,598 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 62.18 2021-05-17 03:18:57,776 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 65.8 2021-05-17 03:19:04,361 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 67.74 2021-05-17 03:19:09,483 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 74.6 2021-05-17 03:19:14,622 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 78.86 2021-05-17 03:19:19,742 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 82.11 2021-05-17 03:19:24,851 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 85.86 2021-05-17 03:19:29,957 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 89.35 2021-05-17 03:19:35,088 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 92.69 2021-05-17 03:19:40,167 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 95.36 2021-05-17 03:19:45,262 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 97.76 2021-05-17 03:19:51,401 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 03:20:01,496 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:20:01,533 | test | INFO | pool-12-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 173.223999977 sec 2021-05-17 03:20:02,522 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:20:02,523 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:20:16,153 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:20:16,153 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:20:36,926 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:20:36,926 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:21:45,138 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1400001049 seconds to complete 2021-05-17 03:21:45,153 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1027.066s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 10 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 11, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:21:57,653 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:21:59,525 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:22:01,632 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6024 type=client open_count=1 channel=[id: 0x6e664d4d, 0.0.0.0/0.0.0.0:48131] timeout=300.0>) 2021-05-17 03:22:25,556 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6025 type=client open_count=1 channel=[id: 0xbb56cd7e, 0.0.0.0/0.0.0.0:60859] timeout=300.0>) 2021-05-17 03:22:27,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6026 type=client open_count=1 channel=[id: 0xc18f2465, 0.0.0.0/0.0.0.0:37907] timeout=300.0>) 2021-05-17 03:22:29,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6027 type=client open_count=1 channel=[id: 0x8b7f8d9f, 0.0.0.0/0.0.0.0:48404] timeout=300.0>) 2021-05-17 03:22:54,755 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6028 type=client open_count=1 channel=[id: 0x290b65f5, 0.0.0.0/0.0.0.0:59097] timeout=300.0>) 2021-05-17 03:22:56,798 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6029 type=client open_count=1 channel=[id: 0xab9ac5a6, 0.0.0.0/0.0.0.0:47605] timeout=300.0>) 2021-05-17 03:22:58,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602a type=client open_count=1 channel=[id: 0xa9543501, 0.0.0.0/0.0.0.0:41979] timeout=300.0>) 2021-05-17 03:23:25,428 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602b type=client open_count=1 channel=[id: 0x0f316aa7, 0.0.0.0/0.0.0.0:40947] timeout=300.0>) 2021-05-17 03:23:27,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602c type=client open_count=1 channel=[id: 0x2c6f9bdd, 0.0.0.0/0.0.0.0:59938] timeout=300.0>) 2021-05-17 03:23:29,493 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602d type=client open_count=1 channel=[id: 0xa69678e5, 0.0.0.0/0.0.0.0:50287] timeout=300.0>) 2021-05-17 03:23:55,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602e type=client open_count=1 channel=[id: 0x395730ed, 0.0.0.0/0.0.0.0:46932] timeout=300.0>) 2021-05-17 03:23:57,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602f type=client open_count=1 channel=[id: 0xe624228d, 0.0.0.0/0.0.0.0:35234] timeout=300.0>) 2021-05-17 03:23:59,265 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6030 type=client open_count=1 channel=[id: 0x0a24ccfc, 0.0.0.0/0.0.0.0:44145] timeout=300.0>) 2021-05-17 03:24:26,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6031 type=client open_count=1 channel=[id: 0x7f9f1660, 0.0.0.0/0.0.0.0:46700] timeout=300.0>) 2021-05-17 03:24:28,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6032 type=client open_count=1 channel=[id: 0x83d8b775, 0.0.0.0/0.0.0.0:43378] timeout=300.0>) 2021-05-17 03:24:30,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:24:51,571 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:25:08,979 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:25:09,055 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:25:09,197 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:25:23,779 | test | INFO | pool-13-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.23005383423 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:25:23,849 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:25:34,015 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:25:34,051 | test | INFO | pool-13-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2709999084 sec 2021-05-17 03:25:34,437 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:26:29,555 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:26:29,898 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 685582936 | 161103560 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 302167600 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 836786712 | 92346589 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 03:26:29,900 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:27:20,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:27:20,614 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:27:42,378 | test | INFO | pool-13-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 140.792 | | 114.856 | ................................................................................X | 114.856 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:27:42,549 | test | INFO | pool-13-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................................................................................................**..X | 122.468 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:27:42,819 | test | INFO | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................**................................................................X | 25623.271 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:27:43,108 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9732078568 | 6437847040 | 4172472320 | 2597888000 / 7511588864 | 71649 / 214121 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.7896897503 | 6437847040 | 4108689408 | 2670399488 / 7511588864 | 69525 / 184675 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9461596702 | 6437847040 | 4104855552 | 2659078144 / 7511588864 | 71754 / 219935 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 97.4006955885 | 6437847040 | 3806674944 | 3139280896 / 9530753024 | 71581 / 145178 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9733754888 | 6437847040 | 4141879296 | 2644598784 / 7511588864 | 70887 / 175996 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:27:58,795 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:29:26,259 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211874840 | 319990167 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319636832 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 836021960 | 619601592 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:29:26,259 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:29:36,523 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 03:29:38,176 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 03:29:38,177 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_recovery 2021-05-17 03:29:38,177 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:29:47,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:29:47,546 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:29:47,548 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:29:58,151 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket2:q7GNTf-34-521000:tM17GyXfKDR2-34-523000 +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:00,816 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in default:scope1:Wzooi-34-545000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-436 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,459 | test | INFO | pool-13-thread-15 | [table_view:display:72] Keys failed in default:_default:4XNodbPcjHD-34-543000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,480 | test | INFO | pool-13-thread-7 | [table_view:display:72] Keys failed in default:fALratH-34-553000:E3a5YEUM45lDcfbakQMO-34-555000 +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-16 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,849 | test | INFO | pool-13-thread-27 | [table_view:display:72] Keys failed in default:scope1:Wzooi-34-545000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-926 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-904 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-966 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-901 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-930 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-970 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,865 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in default:fALratH-34-553000:LQ-34-553000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-667 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-699 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-674 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-690 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:02,388 | test | INFO | pool-13-thread-11 | [table_view:display:72] Keys failed in default:scope1:OrDK1mLiJa8Ns5-34-546000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-709 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-808 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-847 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-826 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-749 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-848 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-804 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-724 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-802 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-887 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-700 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-722 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-767 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-745 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-888 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-841 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-863 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-786 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-783 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-882 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-761 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-817 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-839 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-737 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-738 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-879 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-854 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-799 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-778 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-811 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-852 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-731 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-795 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-894 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-774 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-870 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-891 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-790 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:03,088 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 03:30:03,092 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 19 docs from 0 to 500 2021-05-17 03:30:03,094 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 41 docs from 0 to 500 2021-05-17 03:30:03,094 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 38 docs from 500 to 980 2021-05-17 03:30:03,104 | test | INFO | pool-13-thread-2 | [table_view:display:72] Keys failed in default:fALratH-34-553000:T5hcTCsaKMV%X-34-554000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-667 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-699 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-674 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-690 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:03,381 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 45 docs from 500 to 980 2021-05-17 03:30:03,382 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-17 03:30:03,384 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 3 docs from 0 to 500 2021-05-17 03:30:03,384 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 45 docs from 500 to 980 2021-05-17 03:30:12,525 | test | INFO | pool-13-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.**...................................................................................................X | 5789.645 | | 4747.162 | ................................................................................X | 4747.162 | | 1236.179 | ....................*X | 1236.179 | | 61.817 | X | 61.817 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:30:12,890 | test | INFO | pool-13-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 4058.789 | | 163.608 | ...**....X | 356.264 | +---------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:30:12,933 | test | INFO | pool-13-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..............................................................................*.....**.................X | 19342.030 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:30:15,664 | test | INFO | pool-13-thread-5 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.25499987602 seconds to complete 2021-05-17 03:30:15,670 | test | INFO | pool-13-thread-5 | [common_lib:sleep:22] Sleep 8.25499987602 seconds. Reason: Wait after rebalance complete 2021-05-17 03:32:23,992 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0640001296997 seconds 2021-05-17 03:32:36,191 | test | INFO | pool-13-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.48099994659 seconds to complete 2021-05-17 03:32:36,197 | test | INFO | pool-13-thread-24 | [common_lib:sleep:22] Sleep 8.48099994659 seconds. Reason: Wait after rebalance complete 2021-05-17 03:34:44,734 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0539999008179 seconds 2021-05-17 03:34:44,736 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:34:44,736 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:34:50,586 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:35:05,710 | test | INFO | pool-13-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 4248.947 | | 2778.516 | ................................................................X | 2778.516 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:35:07,174 | test | INFO | pool-13-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..........................................................................*..........................X | 6734.454 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:35:07,898 | test | INFO | pool-13-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................................*.................................*..............................X | 22955.942 | | 22928.688 | ..................................................................................................X | 22928.688 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:35:08,000 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,000 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,055 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,055 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:35:23,428 | test | INFO | pool-13-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.516683889463 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 67.7646980135 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 67.9155971866 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 78.3710588786 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 2.86990433652 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:35:23,453 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:28,510 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:33,569 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:38,628 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:43,711 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:48,907 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.36 2021-05-17 03:35:54,055 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 7.21 2021-05-17 03:35:59,186 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 13.65 2021-05-17 03:36:04,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.92 2021-05-17 03:36:09,650 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 20.3 2021-05-17 03:36:14,720 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.58 2021-05-17 03:36:19,782 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 03:36:24,852 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.4 2021-05-17 03:36:29,924 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.04 2021-05-17 03:36:35,003 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.77 2021-05-17 03:36:40,075 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 47.24 2021-05-17 03:36:45,187 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 49.38 2021-05-17 03:36:50,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 53.33 2021-05-17 03:36:55,415 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 58.49 2021-05-17 03:37:02,173 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.9 2021-05-17 03:37:07,302 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 73.14 2021-05-17 03:37:12,372 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.17 2021-05-17 03:37:18,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-17 03:37:28,405 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:37:28,443 | test | INFO | pool-13-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 125.015000105 sec 2021-05-17 03:37:28,444 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 03:37:30,443 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 03:37:31,430 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:37:31,431 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:37:44,351 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:37:44,352 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:38:04,990 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:38:04,990 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:39:03,371 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.128000021 seconds to complete 2021-05-17 03:39:03,387 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1038.075s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 11 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 12, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:39:12,762 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:39:13,441 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:39:13,861 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8a type=client open_count=1 channel=[id: 0x472be428, 0.0.0.0/0.0.0.0:49472] timeout=300.0>) 2021-05-17 03:39:34,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8b type=client open_count=1 channel=[id: 0x803d9297, 0.0.0.0/0.0.0.0:52063] timeout=300.0>) 2021-05-17 03:39:36,693 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8c type=client open_count=1 channel=[id: 0x4a0d19be, 0.0.0.0/0.0.0.0:48917] timeout=300.0>) 2021-05-17 03:39:38,713 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8d type=client open_count=1 channel=[id: 0xbeb90def, 0.0.0.0/0.0.0.0:52125] timeout=300.0>) 2021-05-17 03:40:03,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8e type=client open_count=1 channel=[id: 0x302f9760, 0.0.0.0/0.0.0.0:43281] timeout=300.0>) 2021-05-17 03:40:05,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8f type=client open_count=1 channel=[id: 0xd3dfedfe, 0.0.0.0/0.0.0.0:34933] timeout=300.0>) 2021-05-17 03:40:07,142 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff90 type=client open_count=1 channel=[id: 0x264ae60c, 0.0.0.0/0.0.0.0:43863] timeout=300.0>) 2021-05-17 03:40:31,665 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff91 type=client open_count=1 channel=[id: 0x278def2f, 0.0.0.0/0.0.0.0:52322] timeout=300.0>) 2021-05-17 03:40:33,691 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff92 type=client open_count=1 channel=[id: 0x322c925f, 0.0.0.0/0.0.0.0:50279] timeout=300.0>) 2021-05-17 03:40:35,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff93 type=client open_count=1 channel=[id: 0x706f7bc7, 0.0.0.0/0.0.0.0:34242] timeout=300.0>) 2021-05-17 03:41:01,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff94 type=client open_count=1 channel=[id: 0x7e95b8f6, 0.0.0.0/0.0.0.0:38346] timeout=300.0>) 2021-05-17 03:41:03,928 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff95 type=client open_count=1 channel=[id: 0xc79a21ba, 0.0.0.0/0.0.0.0:46609] timeout=300.0>) 2021-05-17 03:41:05,956 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff96 type=client open_count=1 channel=[id: 0x9c6b42a8, 0.0.0.0/0.0.0.0:53922] timeout=300.0>) 2021-05-17 03:41:35,891 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff97 type=client open_count=1 channel=[id: 0x3c3db1e6, 0.0.0.0/0.0.0.0:41188] timeout=300.0>) 2021-05-17 03:41:37,920 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff98 type=client open_count=1 channel=[id: 0x79201f28, 0.0.0.0/0.0.0.0:46867] timeout=300.0>) 2021-05-17 03:41:39,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:42:04,108 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:42:20,650 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:42:20,690 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:42:20,762 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:42:34,428 | test | INFO | pool-14-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.30666666667 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:42:34,473 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:42:44,595 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:42:44,641 | test | INFO | pool-14-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2120001316 sec 2021-05-17 03:42:45,061 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:43:32,223 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:43:32,674 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 946590824 | 110992556 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 284335920 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1380384840 | 51620187 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 03:43:32,676 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:44:15,217 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:44:15,227 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:44:36,875 | test | INFO | pool-14-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 24143.210 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:44:38,726 | test | INFO | pool-14-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....................................................................**..............................X | 176.725 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:44:40,346 | test | INFO | pool-14-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..**..................................................................................................*X | 291.046 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:44:40,601 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.6355246439 | 6437847040 | 4209774592 | 2546814976 / 7511588864 | 100582 / 256259 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.791992545 | 6437847040 | 4168409088 | 2590584832 / 7511588864 | 100135 / 203771 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8433307222 | 6437847040 | 4163166208 | 2591281152 / 7511588864 | 98983 / 242447 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9750378592 | 6437847040 | 3679748096 | 3278614528 / 9530753024 | 101211 / 295959 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7910118371 | 6437847040 | 4147585024 | 2629996544 / 7511588864 | 100565 / 258036 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:44:53,809 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:46:21,121 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212938088 | 293539522 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319628640 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 836209640 | 604294682 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:46:21,121 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:46:31,273 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 03:46:31,275 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:46:31,276 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:46:43,181 | test | INFO | pool-14-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.20299983025 seconds to complete 2021-05-17 03:46:43,187 | test | INFO | pool-14-thread-26 | [common_lib:sleep:22] Sleep 8.20299983025 seconds. Reason: Wait after rebalance complete 2021-05-17 03:48:51,448 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 03:49:02,905 | test | INFO | pool-14-thread-8 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.24199986458 seconds to complete 2021-05-17 03:49:02,911 | test | INFO | pool-14-thread-8 | [common_lib:sleep:22] Sleep 8.24199986458 seconds. Reason: Wait after rebalance complete 2021-05-17 03:51:11,213 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 03:51:11,214 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:51:11,216 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:51:18,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:51:32,852 | test | INFO | pool-14-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 11143.727 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:51:34,207 | test | INFO | pool-14-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 3612.957 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:51:34,420 | test | INFO | pool-14-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 2574.382 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:51:34,528 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,585 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,585 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:51:49,523 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:51:49,914 | test | INFO | pool-14-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 1.015 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 78.8049339445 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 69.9811380594 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 86.8874659355 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.521649278357 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:51:50,039 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:51:55,167 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:51:56,000 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:52:00,651 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:06,823 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:11,358 | test | INFO | pool-14-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 13046.518 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:52:11,428 | test | INFO | pool-14-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5236.475 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:52:11,496 | test | INFO | pool-14-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 4891.964 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:52:11,963 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:17,540 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:22,801 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:27,875 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.43 2021-05-17 03:52:33,002 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.75 2021-05-17 03:52:38,084 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 5.79 2021-05-17 03:52:43,207 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 9.34 2021-05-17 03:52:48,365 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 14.01 2021-05-17 03:52:53,540 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 18.55 2021-05-17 03:52:58,753 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.46 2021-05-17 03:53:03,884 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 03:53:09,023 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 28.98 2021-05-17 03:53:14,197 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 34.87 2021-05-17 03:53:19,299 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.05 2021-05-17 03:53:24,424 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.26 2021-05-17 03:53:29,538 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.19 2021-05-17 03:53:34,690 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 51.81 2021-05-17 03:53:41,391 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 55.4 2021-05-17 03:53:46,532 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 62.56 2021-05-17 03:53:51,642 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.84 2021-05-17 03:53:56,828 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 74.8 2021-05-17 03:54:01,928 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 78.75 2021-05-17 03:54:12,028 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:54:12,068 | test | INFO | pool-14-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 142.154999971 sec 2021-05-17 03:54:13,066 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:54:13,068 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:54:29,040 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:54:29,042 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:54:49,084 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:54:49,084 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:55:41,694 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1229999065 seconds to complete 2021-05-17 03:55:41,710 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 998.305s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 12 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 13, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:55:51,174 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:55:51,875 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:55:52,308 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183aa type=client open_count=1 channel=[id: 0xc6159f2c, 0.0.0.0/0.0.0.0:50670] timeout=300.0>) 2021-05-17 03:56:13,062 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183ab type=client open_count=1 channel=[id: 0x197aaf49, 0.0.0.0/0.0.0.0:39074] timeout=300.0>) 2021-05-17 03:56:15,078 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183ac type=client open_count=1 channel=[id: 0xd49c0389, 0.0.0.0/0.0.0.0:42013] timeout=300.0>) 2021-05-17 03:56:17,095 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183ad type=client open_count=1 channel=[id: 0x0ebae3ab, 0.0.0.0/0.0.0.0:47745] timeout=300.0>) 2021-05-17 03:56:41,036 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183ae type=client open_count=1 channel=[id: 0xe01f95b7, 0.0.0.0/0.0.0.0:36990] timeout=300.0>) 2021-05-17 03:56:43,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183af type=client open_count=1 channel=[id: 0xdcd09b19, 0.0.0.0/0.0.0.0:41551] timeout=300.0>) 2021-05-17 03:56:45,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b0 type=client open_count=1 channel=[id: 0x79965bac, 0.0.0.0/0.0.0.0:45676] timeout=300.0>) 2021-05-17 03:57:09,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b1 type=client open_count=1 channel=[id: 0xae642b6c, 0.0.0.0/0.0.0.0:36085] timeout=300.0>) 2021-05-17 03:57:11,638 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b2 type=client open_count=1 channel=[id: 0x825a5e3e, 0.0.0.0/0.0.0.0:55391] timeout=300.0>) 2021-05-17 03:57:13,655 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b3 type=client open_count=1 channel=[id: 0xf0cbba97, 0.0.0.0/0.0.0.0:59638] timeout=300.0>) 2021-05-17 03:57:37,671 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b4 type=client open_count=1 channel=[id: 0x0d4f207a, 0.0.0.0/0.0.0.0:52352] timeout=300.0>) 2021-05-17 03:57:39,691 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b5 type=client open_count=1 channel=[id: 0x36382df1, 0.0.0.0/0.0.0.0:36026] timeout=300.0>) 2021-05-17 03:57:41,709 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b6 type=client open_count=1 channel=[id: 0xda666016, 0.0.0.0/0.0.0.0:45077] timeout=300.0>) 2021-05-17 03:58:05,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b7 type=client open_count=1 channel=[id: 0xfca74b30, 0.0.0.0/0.0.0.0:58847] timeout=300.0>) 2021-05-17 03:58:07,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b8 type=client open_count=1 channel=[id: 0xf7b71b6d, 0.0.0.0/0.0.0.0:41192] timeout=300.0>) 2021-05-17 03:58:09,609 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:58:26,941 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:58:40,677 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:58:40,698 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:58:40,734 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:58:54,697 | test | INFO | pool-15-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.08149728342 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:58:54,720 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:59:04,792 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:59:04,822 | test | INFO | pool-15-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1259999275 sec 2021-05-17 03:59:05,072 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:59:50,569 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:59:50,859 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 627925672 | 98651537 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 211980088 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 562588744 | 79934784 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 03:59:50,861 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:00:33,388 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:00:33,397 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:01:13,168 | test | INFO | pool-15-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.........**...........................................................................................*X | 13531.578 | | 13288.278 | .................................................................................................X | 13288.278 | | 13261.156 | .................................................................................................*X | 13261.156 | | 12716.754 | ............................................................................................X | 12716.754 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:01:14,589 | test | INFO | pool-15-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.**...**..........................*.**.....................................................................*X | 154.514 | +-------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:01:14,867 | test | INFO | pool-15-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...................................................................................**.....**X | 69.625 | | 55.598 | .....................................................................**..............................*X | 78.707 | +--------+---------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 04:01:15,125 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.5526473927 | 6437847040 | 4255506432 | 2479099904 / 7511588864 | 83463 / 165643 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.9488119417 | 6437847040 | 4221444096 | 2515341312 / 7511588864 | 81385 / 162604 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.87 | 6437847040 | 4228124672 | 2502098944 / 7511588864 | 82728 / 167418 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.4014278787 | 6437847040 | 3734384640 | 3193126912 / 9530753024 | 83178 / 163179 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.5051160302 | 6437847040 | 4281110528 | 2475749376 / 7511588864 | 82364 / 165807 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:01:28,927 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:02:55,592 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162373384 | 241888316 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 245360528 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 609616504 | 405183413 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:02:55,595 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 04:03:05,671 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 04:03:05,673 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 04:03:05,674 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:03:48,188 | test | INFO | pool-15-thread-22 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 42.3129999638 seconds to complete 2021-05-17 04:03:48,194 | test | INFO | pool-15-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 04:05:53,250 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0529999732971 seconds 2021-05-17 04:05:53,250 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:05:53,252 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:05:59,601 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:06:31,973 | test | INFO | pool-15-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.........................................**..............*..........................................**...*X | 12877.411 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:06:33,095 | test | INFO | pool-15-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*..................................**........................................................**X | 2125.554 | | 46.009 | .*X | 46.009 | +--------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:06:34,223 | test | INFO | pool-15-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*.............................**.............................................................**X | 2413.283 | | 234.050 | ........**X | 234.050 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:06:34,470 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:06:34,526 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:06:45,240 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:06:45,415 | test | INFO | pool-15-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 4.92016400547 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 99.4000099998 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 51.1217199706 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 59.5319843995 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 52.7074548758 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 04:06:45,453 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:06:51,194 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:06:52,068 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:06:54,147 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,148 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,155 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,157 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,157 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,158 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:55,999 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,000 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,160 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,161 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,171 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,173 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,408 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1539 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,411 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1539 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,437 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 1.46 2021-05-17 04:06:57,918 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3356 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:57,920 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3356 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,546 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3157 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,548 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3157 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,851 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3364 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,852 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3364 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,993 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,994 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,003 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3562 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,005 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3562 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,263 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,265 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,364 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,365 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,372 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,374 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,424 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,426 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,733 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,734 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,740 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,740 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,765 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,766 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,045 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,046 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,049 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,049 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,053 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,055 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,329 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,331 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:01,663 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.49 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 10 docs from 0 to 5000 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 6 docs from 0 to 5000 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 10 docs from 0 to 5000 2021-05-17 04:07:02,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1266 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:02,885 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1514 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1484 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:03,091 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1645 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:03,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 8 docs from 0 to 5000 2021-05-17 04:07:04,305 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,305 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,378 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1279 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,378 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1279 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,644 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,644 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,178 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,180 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-897 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,792 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2419 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,793 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2419 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,128 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,130 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1646 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1638 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,275 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2800 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2990 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,276 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2963 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2800 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2990 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,543 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3094 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3067 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3104 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,545 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3094 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3067 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3104 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,782 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 3.52 2021-05-17 04:07:07,346 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3602 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3792 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:07,348 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3602 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3792 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,252 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2832 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,252 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2832 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,674 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,676 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,868 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 12 docs from 0 to 5000 2021-05-17 04:07:08,872 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 5 docs from 1000 to 1960 2021-05-17 04:07:09,441 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3949 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3915 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3885 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,443 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3949 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3915 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3893 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3885 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,710 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4187 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4017 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,711 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4187 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4017 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,267 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,269 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4673 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 19 docs from 0 to 5000 2021-05-17 04:07:12,111 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 4.55 2021-05-17 04:07:14,490 | test | INFO | pool-15-thread-13 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:awM3zYY-57-714000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1518 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1488 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:15,204 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 04:07:16,588 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,588 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,822 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-447 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,822 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-447 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,849 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:awM3zYY-57-714000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-335 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,924 | test | INFO | pool-15-thread-21 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5635 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5618 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5788 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,926 | test | INFO | pool-15-thread-21 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5635 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5618 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:17,260 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 5.86 2021-05-17 04:07:18,556 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:18,558 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:19,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:19,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,632 | test | INFO | pool-15-thread-12 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:kur163WM_Tne-57-990000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1626 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,671 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4045 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,671 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4045 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,913 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4219 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4389 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,915 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4219 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4389 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,285 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4617 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4787 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,286 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4617 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,427 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 9 docs from 0 to 5000 2021-05-17 04:07:22,430 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 3 docs from 5000 to 5800 2021-05-17 04:07:22,431 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 04:07:22,443 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 04:07:22,553 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 6.94 2021-05-17 04:07:24,421 | test | INFO | pool-15-thread-7 | [table_view:display:72] Keys failed in bucket1:scope2:l8lbEc7j62ZBQ7qJ-5-134000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:24,701 | test | INFO | pool-15-thread-27 | [table_view:display:72] Keys failed in bucket1:_default:ryf37NkjR9%G-5-128000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:25,785 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 04:07:25,911 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 04:07:27,846 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 8.36 2021-05-17 04:07:29,107 | test | INFO | pool-15-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...................**.................................................................................*X | 11691.578 | | 10062.500 | .....................................................................................*X | 10062.500 | | 9391.399 | ...............................................................................X | 9391.399 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:07:29,236 | test | INFO | pool-15-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *............**........................................................................................X | 2489.068 | | 2489.068 | ..................................................................................................X | 2489.068 | | 1146.803 | .............................................*X | 1146.803 | | 1083.434 | ..........................................X | 1083.434 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:07:31,223 | test | INFO | pool-15-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......**.............................................................................................*X | 2583.773 | | 2209.022 | ....................................................................................*X | 2209.022 | | 1337.462 | ..................................................X | 1337.462 | | 1028.967 | ......................................X | 1028.967 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:07:32,948 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 11.28 2021-05-17 04:07:38,055 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 14.54 2021-05-17 04:07:43,131 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 19.7 2021-05-17 04:07:48,230 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 26.7 2021-05-17 04:07:53,313 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 04:07:58,384 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.06 2021-05-17 04:08:03,500 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 49.73 2021-05-17 04:08:08,611 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 59.66 2021-05-17 04:08:13,973 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.78 2021-05-17 04:08:19,088 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 74.32 2021-05-17 04:08:24,200 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 81.93 2021-05-17 04:08:29,285 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 89.58 2021-05-17 04:08:34,391 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 96.85 2021-05-17 04:08:44,493 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:08:44,625 | test | INFO | pool-15-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 119.209000111 sec 2021-05-17 04:08:44,673 | test | WARNING | Thread-43449 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:ryf37NkjR9%G-5-128000 update failed keys from task: [u'test_collections-6'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43453 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 create failed keys from task: [u'test_collections-3380', u'test_collections-4052', u'test_collections-211', u'test_collections-1678', u'test_collections-381', u'test_collections-3210'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43454 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 create failed keys from task: [u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-1539', u'test_collections-4133', u'test_collections-4035', u'test_collections-3364', u'test_collections-3157'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43455 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 create failed keys from task: [u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-211', u'test_collections-1678', u'test_collections-4133', u'test_collections-381', u'test_collections-4035', u'test_collections-3562', u'test_collections-3356'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43456 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 create failed keys from task: [u'test_collections-3380', u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-211', u'test_collections-1678', u'test_collections-4133', u'test_collections-381', u'test_collections-4035', u'test_collections-3210'] 2021-05-17 04:08:44,677 | test | WARNING | Thread-43459 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 create failed keys from task: [u'test_collections-5635', u'test_collections-5618', u'test_collections-5788'] 2021-05-17 04:08:44,677 | test | WARNING | Thread-43458 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:K9hJ4pRnsUnNqO%q-57-888000:kur163WM_Tne-57-990000 update failed keys from task: [u'test_collections-1741', u'test_collections-1626'] 2021-05-17 04:08:44,687 | test | WARNING | Thread-43457 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 update failed keys from task: [u'test_collections-1249', u'test_collections-1645', u'test_collections-1514', u'test_collections-1484', u'test_collections-1266'] 2021-05-17 04:08:44,721 | test | WARNING | Thread-43460 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:awM3zYY-57-714000 update failed keys from task: [u'test_collections-1518', u'test_collections-1488'] 2021-05-17 04:08:44,743 | test | WARNING | Thread-43460 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:awM3zYY-57-714000 delete failed keys from task: [u'test_collections-318', u'test_collections-335', u'test_collections-288'] 2021-05-17 04:08:44,773 | test | WARNING | Thread-43486 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:mHv6XNug-51-818000 create failed keys from task: [u'test_collections-447', u'test_collections-335', u'test_collections-1059', u'test_collections-4219', u'test_collections-4617', u'test_collections-4045', u'test_collections-2452', u'test_collections-4787', u'test_collections-4389'] 2021-05-17 04:08:44,789 | test | WARNING | Thread-43507 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:l8lbEc7j62ZBQ7qJ-5-134000 update failed keys from task: [u'test_collections-6'] 2021-05-17 04:08:44,792 | test | WARNING | Thread-43529 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 create failed keys from task: [u'test_collections-3094', u'test_collections-3602', u'test_collections-2419', u'test_collections-2990', u'test_collections-3067', u'test_collections-3104', u'test_collections-1279', u'test_collections-2589', u'test_collections-2963', u'test_collections-2800', u'test_collections-3792', u'test_collections-4643'] 2021-05-17 04:08:44,792 | test | WARNING | Thread-43528 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 create failed keys from task: [u'test_collections-4082', u'test_collections-907', u'test_collections-3949', u'test_collections-205', u'test_collections-2832', u'test_collections-1646', u'test_collections-4838', u'test_collections-3903', u'test_collections-4112', u'test_collections-4673', u'test_collections-4017', u'test_collections-3885', u'test_collections-577', u'test_collections-897', u'test_collections-395', u'test_collections-3915', u'test_collections-1638', u'test_collections-4187', u'test_collections-3893'] 2021-05-17 04:08:55,312 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:09:58,482 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 04:09:58,482 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 04:10:10,967 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 04:10:10,967 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 04:10:26,183 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 04:10:26,184 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 04:11:19,997 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1740000248 seconds to complete 2021-05-17 04:11:20,013 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 938.230s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 13 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_14 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_14', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 14, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 04:11:29,443 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:11:30,111 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:11:30,529 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c65 type=client open_count=1 channel=[id: 0x3e52663d, 0.0.0.0/0.0.0.0:38511] timeout=300.0>) 2021-05-17 04:11:51,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c66 type=client open_count=1 channel=[id: 0xe3f7f34f, 0.0.0.0/0.0.0.0:41511] timeout=300.0>) 2021-05-17 04:11:53,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c67 type=client open_count=1 channel=[id: 0x43a8daa1, 0.0.0.0/0.0.0.0:37679] timeout=300.0>) 2021-05-17 04:11:55,341 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c68 type=client open_count=1 channel=[id: 0x2dc8edfc, 0.0.0.0/0.0.0.0:39028] timeout=300.0>) 2021-05-17 04:12:19,454 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c69 type=client open_count=1 channel=[id: 0x853a4dba, 0.0.0.0/0.0.0.0:40667] timeout=300.0>) 2021-05-17 04:12:21,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c6a type=client open_count=1 channel=[id: 0xd34949ed, 0.0.0.0/0.0.0.0:58698] timeout=300.0>) 2021-05-17 04:12:23,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6b type=client open_count=1 channel=[id: 0x859eecf7, 0.0.0.0/0.0.0.0:39930] timeout=300.0>) 2021-05-17 04:12:47,519 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6c type=client open_count=1 channel=[id: 0xec866551, 0.0.0.0/0.0.0.0:54312] timeout=300.0>) 2021-05-17 04:12:49,538 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6d type=client open_count=1 channel=[id: 0xf0e181ed, 0.0.0.0/0.0.0.0:36732] timeout=300.0>) 2021-05-17 04:12:51,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c6e type=client open_count=1 channel=[id: 0x153579a7, 0.0.0.0/0.0.0.0:50064] timeout=300.0>) 2021-05-17 04:13:15,821 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c6f type=client open_count=1 channel=[id: 0x74ef2bb5, 0.0.0.0/0.0.0.0:33732] timeout=300.0>) 2021-05-17 04:13:17,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c70 type=client open_count=1 channel=[id: 0x8506e7ab, 0.0.0.0/0.0.0.0:60279] timeout=300.0>) 2021-05-17 04:13:19,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c71 type=client open_count=1 channel=[id: 0xd5343ac4, 0.0.0.0/0.0.0.0:55888] timeout=300.0>) 2021-05-17 04:13:44,591 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c72 type=client open_count=1 channel=[id: 0x411b3cc5, 0.0.0.0/0.0.0.0:41852] timeout=300.0>) 2021-05-17 04:13:46,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c73 type=client open_count=1 channel=[id: 0xf086346e, 0.0.0.0/0.0.0.0:38017] timeout=300.0>) 2021-05-17 04:13:48,627 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c74 type=client open_count=1 channel=[id: 0x1d0d6ea6, 0.0.0.0/0.0.0.0:39965] timeout=300.0>) 2021-05-17 04:13:50,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 04:14:07,154 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 04:14:20,881 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 04:14:20,903 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:14:20,941 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:14:35,917 | test | INFO | pool-16-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.8434140569 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 04:14:35,941 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:14:46,013 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:14:46,040 | test | INFO | pool-16-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1239998341 sec 2021-05-17 04:14:46,286 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 04:15:31,529 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 04:15:31,792 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 627715080 | 117765415 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 225369296 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 530472984 | 103447262 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 04:15:31,792 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:16:16,030 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:16:16,063 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:16:58,023 | test | INFO | pool-16-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.........................................................................................**..........**.*X | 12351.784 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:16:58,170 | test | INFO | pool-16-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 182.713 | | 25.471 | ............**.............*X | 48.779 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:16:59,586 | test | INFO | pool-16-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.......**............**...........................................**......................................X | 157.407 | +-------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:16:59,931 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.7109154345 | 6437847040 | 4352671744 | 2448977920 / 7511588864 | 97757 / 196365 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.4808047526 | 6437847040 | 4240502784 | 2496221184 / 7511588864 | 96182 / 192500 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.0318649952 | 6437847040 | 4221915136 | 2504466432 / 7511588864 | 96611 / 194324 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 100 | 6437847040 | 3840958464 | 3076243456 / 9530753024 | 96866 / 191250 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.3716980818 | 6437847040 | 4255543296 | 2492289024 / 7511588864 | 96906 / 191023 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:17:14,092 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:18:40,354 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162403336 | 236910036 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 246185040 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 544550472 | 406719618 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:18:40,355 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:18:50,703 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 04:18:50,703 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 04:18:50,704 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:19:01,677 | test | INFO | pool-16-thread-1 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.3789999485 seconds to complete 2021-05-17 04:19:01,683 | test | INFO | pool-16-thread-1 | [common_lib:sleep:22] Sleep 8.3789999485 seconds. Reason: Wait after rebalance complete 2021-05-17 04:21:10,121 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-05-17 04:21:10,121 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:21:10,121 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:21:16,464 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:21:47,653 | test | INFO | pool-16-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.......................................................................X | 9601.421 | | 9601.421 | ......................................................................**.............................**X | 13339.375 | +----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:21:48,930 | test | INFO | pool-16-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...................................................................................................**X | 1728.282 | | 1340.883 | ............................................................................**X | 1340.883 | | 56.522 | ..X | 56.522 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:21:49,555 | test | INFO | pool-16-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................**X | 2222.753 | | 1352.913 | ...........................................................**X | 1352.913 | | 45.335 | .*X | 45.335 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:21:49,832 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:21:49,884 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:22:01,581 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:22:01,786 | test | INFO | pool-16-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 8.51819136348 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 85.5523653724 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 86.5970074396 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 91.9657463359 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 82.5512878295 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 04:22:01,819 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:22:08,605 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:22:08,770 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:22:13,562 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2076 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:13,566 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2076 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:13,780 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 1.61 2021-05-17 04:22:15,223 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,223 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-171 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,309 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3138 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,311 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3138 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,424 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,426 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,358 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1022 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1198 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,361 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1022 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1198 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,611 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4119 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4089 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,611 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4119 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4089 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,617 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1198 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,618 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1198 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,651 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,651 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,808 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-32 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,809 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-32 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,861 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,862 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,871 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,872 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,888 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1402 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,888 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1402 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,398 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,400 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,983 | test | INFO | pool-16-thread-18 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:K6pGTLfDruZK3Z-8-517000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4940 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,984 | test | INFO | pool-16-thread-18 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:K6pGTLfDruZK3Z-8-517000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4893 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4940 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,431 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1812 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1982 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,431 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1812 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1982 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,482 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,483 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,730 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,730 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,733 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,734 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,736 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3126 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,737 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3126 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,780 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,782 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,799 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,805 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,108 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.52 2021-05-17 04:22:19,157 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,158 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,171 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,173 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,190 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,190 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,437 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,438 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-979 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,497 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2496 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,499 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2496 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,503 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,503 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,503 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,505 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,693 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,694 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,775 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2496 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,776 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2496 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,798 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2496 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,799 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2506 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2496 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,802 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3986 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3816 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,802 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3986 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,812 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,812 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,973 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,976 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1333 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,066 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,068 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,125 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,127 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2660 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,530 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3331 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,532 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3331 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,706 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1874 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1984 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,707 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1874 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1984 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1814 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,709 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 04:22:20,710 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 04:22:20,720 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3331 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,720 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3331 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,849 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3423 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,849 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3331 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,851 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3423 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,851 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3351 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3331 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,855 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4965 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4905 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4895 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:20,855 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4965 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4905 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4895 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,101 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3423 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,102 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3423 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,211 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,213 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3778 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,464 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,466 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3778 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,681 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3833 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,683 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,842 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,927 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4194 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4004 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:21,937 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4194 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4004 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,135 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1316 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1286 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,233 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,236 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4207 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4397 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,323 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3423 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,325 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3443 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3423 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,371 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1464 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,489 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,490 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4575 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,631 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3729 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3712 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3682 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,631 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3729 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3712 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3682 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:22,737 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 17 docs from 0 to 5000 2021-05-17 04:22:22,869 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 23 docs from 0 to 5000 2021-05-17 04:22:23,210 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,210 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4136 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,273 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2964 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,276 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2964 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,430 | test | INFO | pool-16-thread-28 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1755 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,566 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3060 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,568 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3060 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,825 | test | INFO | pool-16-thread-8 | [table_view:display:72] Keys failed in bucket1:9r47QgZyKUOuS-14-517000:nw-14-672000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1193 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1027 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1003 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,828 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4660 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4644 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:23,829 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4660 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4644 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:24,029 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 18 docs from 0 to 5000 2021-05-17 04:22:24,515 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 3.43 2021-05-17 04:22:24,687 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4081 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4111 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:24,688 | test | INFO | pool-16-thread-15 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4081 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4111 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:25,200 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4742 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:25,201 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4742 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:25,354 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 15 docs from 0 to 5000 2021-05-17 04:22:25,470 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 10 docs from 0 to 5000 2021-05-17 04:22:29,766 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 4.95 2021-05-17 04:22:30,657 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 5 docs from 1000 to 1960 2021-05-17 04:22:30,658 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 04:22:30,661 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 04:22:32,046 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5293 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5303 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:32,046 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5293 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5303 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:32,368 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:32,369 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:83HC-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5708 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5698 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:32,612 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-304 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-294 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:32,766 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-476 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:34,984 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 6.22 2021-05-17 04:22:36,721 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 4 docs from 5000 to 5800 2021-05-17 04:22:36,723 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 4 docs from 0 to 1000 2021-05-17 04:22:36,869 | test | INFO | pool-16-thread-10 | [table_view:display:72] Keys failed in bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:37,240 | test | INFO | pool-16-thread-10 | [table_view:display:72] Keys failed in bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-768 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:37,651 | test | INFO | pool-16-thread-6 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:37,653 | test | INFO | pool-16-thread-6 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5245 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:38,408 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 2 docs from 0 to 1000 2021-05-17 04:22:38,815 | test | INFO | pool-16-thread-14 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:39,122 | test | INFO | pool-16-thread-14 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-408 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-598 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:39,210 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5657 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:39,210 | test | INFO | pool-16-thread-5 | [table_view:display:72] Keys failed in bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5657 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:39,213 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 04:22:39,466 | test | INFO | pool-16-thread-14 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-841 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:40,085 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5165 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:40,085 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5165 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:40,183 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 7.31 2021-05-17 04:22:40,415 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 04:22:40,417 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 4 docs from 0 to 1000 2021-05-17 04:22:41,415 | test | ERROR | pool-16-thread-1 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 04:22:43,813 | test | INFO | pool-16-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........................................................**............................................*X | 11207.087 | | 10661.318 | ..............................................................................................*X | 10661.318 | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:22:45,328 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 9.99 2021-05-17 04:22:45,720 | test | INFO | pool-16-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...................**.................................................................................*X | 3581.431 | | 679.639 | .................**X | 679.639 | +---------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:22:46,019 | test | INFO | pool-16-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.............................................................**.......................................*X | 2636.175 | | 779.879 | ............................**X | 779.879 | +---------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:22:50,417 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 13.45 2021-05-17 04:22:55,502 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 17.68 2021-05-17 04:23:00,585 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.95 2021-05-17 04:23:05,664 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 30.19 2021-05-17 04:23:10,756 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 34.9 2021-05-17 04:23:15,885 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 43.81 2021-05-17 04:23:21,013 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 53.69 2021-05-17 04:23:26,114 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 63.44 2021-05-17 04:23:31,216 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.77 2021-05-17 04:23:36,295 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 77.22 2021-05-17 04:23:41,371 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 84.88 2021-05-17 04:23:46,441 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 92.66 2021-05-17 04:23:51,540 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 98.6 2021-05-17 04:24:01,637 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:24:01,732 | test | INFO | pool-16-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 119.947000027 sec 2021-05-17 04:24:01,789 | test | WARNING | Thread-47238 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 create failed keys from task: [u'test_collections-4261', u'test_collections-4119', u'test_collections-3138', u'test_collections-4913', u'test_collections-4089', u'test_collections-2076', u'test_collections-4883'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47241 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WTBAjv-14-882000:K6pGTLfDruZK3Z-8-517000 create failed keys from task: [u'test_collections-4913', u'test_collections-4903', u'test_collections-4893', u'test_collections-4940', u'test_collections-4883'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47242 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 create failed keys from task: [u'test_collections-2082', u'test_collections-32', u'test_collections-2274', u'test_collections-2112', u'test_collections-3443', u'test_collections-2496', u'test_collections-3423', u'test_collections-1982', u'test_collections-3778', u'test_collections-1812', u'test_collections-2506', u'test_collections-3351', u'test_collections-2660', u'test_collections-3331', u'test_collections-4742'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47244 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 create failed keys from task: [u'test_collections-2082', u'test_collections-3729', u'test_collections-1802', u'test_collections-2274', u'test_collections-3682', u'test_collections-2112', u'test_collections-3443', u'test_collections-2496', u'test_collections-3423', u'test_collections-4136', u'test_collections-3712', u'test_collections-2506', u'test_collections-3351', u'test_collections-2660', u'test_collections-3331', u'test_collections-4660', u'test_collections-4644', u'test_collections-1992'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47239 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 create failed keys from task: [u'test_collections-2082', u'test_collections-120', u'test_collections-2274', u'test_collections-2112', u'test_collections-3443', u'test_collections-4575', u'test_collections-4207', u'test_collections-3833', u'test_collections-2506', u'test_collections-1370', u'test_collections-3351', u'test_collections-2660', u'test_collections-4004', u'test_collections-1008', u'test_collections-1802', u'test_collections-4397', u'test_collections-2496', u'test_collections-3423', u'test_collections-4194', u'test_collections-3778', u'test_collections-3331', u'test_collections-1198', u'test_collections-1992'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47240 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 update failed keys from task: [u'test_collections-1755'] 2021-05-17 04:24:01,789 | test | WARNING | Thread-47243 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 create failed keys from task: [u'test_collections-3986', u'test_collections-1402', u'test_collections-120', u'test_collections-1008', u'test_collections-1802', u'test_collections-1022', u'test_collections-3126', u'test_collections-4895', u'test_collections-2822', u'test_collections-4965', u'test_collections-4905', u'test_collections-3816', u'test_collections-171', u'test_collections-1370', u'test_collections-1592', u'test_collections-1198', u'test_collections-1992'] 2021-05-17 04:24:01,845 | test | WARNING | Thread-47240 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 delete failed keys from task: [u'test_collections-304', u'test_collections-225', u'test_collections-476', u'test_collections-294'] 2021-05-17 04:24:01,848 | test | WARNING | Thread-47245 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:SNXf%C_142QN-6-481000:L_8lZior13PirolZ_CzX-6-750000 create failed keys from task: [u'test_collections-4081', u'test_collections-3060', u'test_collections-979', u'test_collections-1874', u'test_collections-1984', u'test_collections-2964', u'test_collections-1814', u'test_collections-4111', u'test_collections-1055', u'test_collections-1333'] 2021-05-17 04:24:01,861 | test | WARNING | Thread-47348 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 delete failed keys from task: [u'test_collections-408', u'test_collections-96', u'test_collections-841', u'test_collections-598'] 2021-05-17 04:24:01,865 | test | WARNING | Thread-47240 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WTBAjv-14-882000:YTs84WVRNhkyLJ4VsR-15-15000 create failed keys from task: [u'test_collections-5245'] 2021-05-17 04:24:01,877 | test | WARNING | Thread-47349 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:9r47QgZyKUOuS-14-517000:nw-14-672000 update failed keys from task: [u'test_collections-1193', u'test_collections-1027', u'test_collections-1003'] 2021-05-17 04:24:01,882 | test | WARNING | Thread-47348 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:k2MzwJ--15-156000:hNSGBNNoV9URD6OX%U%4-15-268000 create failed keys from task: [u'test_collections-5165'] 2021-05-17 04:24:01,907 | test | WARNING | Thread-47350 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 delete failed keys from task: [u'test_collections-1', u'test_collections-768'] 2021-05-17 04:24:01,913 | test | WARNING | Thread-47347 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:k2MzwJ--15-156000:83HC-15-268000 update failed keys from task: [u'test_collections-1115', u'test_collections-1316', u'test_collections-1085', u'test_collections-1286', u'test_collections-1464'] 2021-05-17 04:24:01,921 | test | WARNING | Thread-47350 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:9r47QgZyKUOuS-14-517000:o4ABl-lp1Sk8C8LVZQb-14-672000 create failed keys from task: [u'test_collections-5657'] 2021-05-17 04:24:01,948 | test | WARNING | Thread-47347 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:k2MzwJ--15-156000:83HC-15-268000 create failed keys from task: [u'test_collections-5293', u'test_collections-5708', u'test_collections-5303', u'test_collections-5698'] 2021-05-17 04:24:11,775 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:25:12,313 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 04:25:12,313 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 04:25:19,065 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 04:25:19,066 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 04:25:39,726 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 04:25:39,726 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 04:26:35,503 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1270000935 seconds to complete 2021-05-17 04:26:35,517 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 915.484s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 14 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_14 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_15 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_15', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 15, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 04:26:45,032 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #15 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:26:45,720 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #15 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:26:46,134 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8482cf type=client open_count=1 channel=[id: 0x9c3fd2fa, 0.0.0.0/0.0.0.0:42788] timeout=300.0>) 2021-05-17 04:27:06,917 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8482d0 type=client open_count=1 channel=[id: 0x1bbb6a34, 0.0.0.0/0.0.0.0:37099] timeout=300.0>) 2021-05-17 04:27:08,934 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8482d1 type=client open_count=1 channel=[id: 0x27a2a76e, 0.0.0.0/0.0.0.0:38640] timeout=300.0>) 2021-05-17 04:27:10,953 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8482d2 type=client open_count=1 channel=[id: 0x590cece3, 0.0.0.0/0.0.0.0:55820] timeout=300.0>) 2021-05-17 04:27:34,888 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8482d3 type=client open_count=1 channel=[id: 0x81d5eddc, 0.0.0.0/0.0.0.0:52020] timeout=300.0>) 2021-05-17 04:27:36,907 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8482d4 type=client open_count=1 channel=[id: 0xa70375e6, 0.0.0.0/0.0.0.0:57756] timeout=300.0>) 2021-05-17 04:27:38,924 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8482d5 type=client open_count=1 channel=[id: 0x91b0caf7, 0.0.0.0/0.0.0.0:46975] timeout=300.0>) 2021-05-17 04:28:02,786 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8482d6 type=client open_count=1 channel=[id: 0x7c2e169f, 0.0.0.0/0.0.0.0:53691] timeout=300.0>) 2021-05-17 04:28:04,805 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8482d7 type=client open_count=1 channel=[id: 0xe3997d98, 0.0.0.0/0.0.0.0:50903] timeout=300.0>) 2021-05-17 04:28:06,822 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8482d8 type=client open_count=1 channel=[id: 0x42711e48, 0.0.0.0/0.0.0.0:37844] timeout=300.0>) 2021-05-17 04:28:30,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8482d9 type=client open_count=1 channel=[id: 0x96a42e74, 0.0.0.0/0.0.0.0:60663] timeout=300.0>) 2021-05-17 04:28:32,977 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8482da type=client open_count=1 channel=[id: 0xd0a6cfa3, 0.0.0.0/0.0.0.0:34712] timeout=300.0>) 2021-05-17 04:28:34,996 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8482db type=client open_count=1 channel=[id: 0xa015fe26, 0.0.0.0/0.0.0.0:60193] timeout=300.0>) 2021-05-17 04:28:59,349 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8482dc type=client open_count=1 channel=[id: 0xfa92ef95, 0.0.0.0/0.0.0.0:39363] timeout=300.0>) 2021-05-17 04:29:01,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8482dd type=client open_count=1 channel=[id: 0xd908076b, 0.0.0.0/0.0.0.0:43881] timeout=300.0>) 2021-05-17 04:29:03,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 04:29:20,512 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 04:29:34,183 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 04:29:34,211 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #15 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:29:34,249 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #15 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:29:48,282 | test | INFO | pool-17-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.938333333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 04:29:48,305 | test | INFO | pool-17-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:29:58,381 | test | INFO | pool-17-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:29:58,420 | test | INFO | pool-17-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1390001774 sec 2021-05-17 04:29:58,676 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 04:30:43,507 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 04:30:43,802 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 589531160 | 103845989 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 216034624 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 584669944 | 71959148 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 04:30:43,803 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:31:28,040 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:31:28,051 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:32:11,329 | test | INFO | pool-17-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...........................................**.........................................................*X | 13245.992 | | 12472.079 | .............................................................................................*X | 12472.079 | | 12044.194 | .........................................................................................X | 12044.194 | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:32:12,730 | test | INFO | pool-17-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.**........................................**...........................................................*X | 129.584 | | 100.455 | ............................................................................X | 100.455 | | 98.545 | ...........................................................................X | 98.545 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:32:13,288 | test | INFO | pool-17-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.................................................**............................................*...**....*X | 109.149 | +-------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:32:13,575 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.8968300191 | 6437847040 | 4326363136 | 2469519360 / 7511588864 | 89066 / 178897 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.3493418535 | 6437847040 | 4247244800 | 2486272000 / 7511588864 | 86738 / 172936 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.347567214 | 6437847040 | 4195807232 | 2525335552 / 7511588864 | 88996 / 178932 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 95.7666649947 | 6437847040 | 3797954560 | 3133431808 / 9530753024 | 88259 / 171928 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.6622694822 | 6437847040 | 4235780096 | 2522210304 / 7511588864 | 88257 / 177693 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:32:27,417 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:33:53,687 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162163304 | 244949913 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 246155040 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 579335544 | 406839241 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:33:53,687 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #15 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:34:03,750 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 04:34:03,752 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 04:34:03,753 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:34:43,259 | test | INFO | pool-17-thread-8 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 39.3029999733 seconds to complete 2021-05-17 04:34:43,265 | test | INFO | pool-17-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 04:36:48,325 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 04:36:48,325 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:36:48,326 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:36:55,515 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:37:18,226 | test | INFO | pool-17-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.........................................................................*...........................*X | 8897.640 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:37:19,497 | test | INFO | pool-17-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..........................................................................*..........................*X | 2912.673 | | 224.883 | ......X | 224.883 | +---------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:37:19,805 | test | INFO | pool-17-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................**X | 2511.555 | | 1634.299 | ................................................................X | 1634.299 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:37:20,059 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:37:20,115 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:37:32,095 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:37:32,265 | test | INFO | pool-17-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 1.82006066869 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 90.4817547491 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 91.5740120563 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 91.6368440416 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 55.206093638 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 04:37:32,331 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:37:37,447 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:37:38,961 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:37:42,707 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:37:47,937 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:37:52,052 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3961 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,053 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3961 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,855 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,858 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,913 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4652 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,913 | test | INFO | pool-17-thread-24 | [table_view:display:72] Keys failed in bucket1:QKDoDq2U12XgBps5rfq-53-153000:cjG9YkgZ5zCmfnB-53-288000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1743 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:52,914 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4652 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,145 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5346 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,147 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5346 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,161 | test | INFO | pool-17-thread-24 | [table_view:display:72] Keys failed in bucket1:QKDoDq2U12XgBps5rfq-53-153000:cjG9YkgZ5zCmfnB-53-288000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,233 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,234 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,407 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5471 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5434 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:53,408 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5471 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5434 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:54,036 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in bucket1:v6Wo%-53-811000:9-53-911000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1667 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:54,203 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.29 2021-05-17 04:37:54,674 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:54,676 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-933 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:55,700 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 04:37:55,838 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2255 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:55,838 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2255 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:57,062 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3409 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3599 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:57,062 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3528 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3409 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3599 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:58,509 | test | INFO | pool-17-thread-1 | [table_view:display:72] Keys failed in bucket1:QKDoDq2U12XgBps5rfq-53-153000:52-53-287000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5532 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:58,509 | test | INFO | pool-17-thread-1 | [table_view:display:72] Keys failed in bucket1:QKDoDq2U12XgBps5rfq-53-153000:52-53-287000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:37:59,568 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 2.05 2021-05-17 04:38:00,946 | test | INFO | pool-17-thread-20 | [table_view:display:72] Keys failed in bucket1:_default:g%Wy1BIfdUC-58-736000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:38:01,082 | test | INFO | pool-17-thread-6 | [table_view:display:72] Keys failed in bucket1:_default:uOQIJLfZbdrf-58-736000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:38:01,104 | test | INFO | pool-17-thread-20 | [table_view:display:72] Keys failed in bucket1:_default:152G-58-736000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:38:01,453 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 04:38:01,456 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 4 docs from 5000 to 5800 2021-05-17 04:38:01,457 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 04:38:01,460 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 04:38:01,469 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 04:38:03,019 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in bucket1:v6Wo%-53-811000:9-53-911000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5148 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:38:03,019 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in bucket1:v6Wo%-53-811000:9-53-911000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5148 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:38:05,055 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 3.52 2021-05-17 04:38:07,861 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 04:38:07,861 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 04:38:07,862 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 04:38:07,865 | test | ERROR | pool-17-thread-13 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 04:38:10,059 | test | INFO | pool-17-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.....................................**...............................................................X | 9114.046 | | 8231.387 | .........................................................................................X | 8231.387 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:38:10,404 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 6.95 2021-05-17 04:38:12,265 | test | INFO | pool-17-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...............................................................................................*..*...X | 2531.674 | | 1716.781 | ..................................................................X | 1716.781 | | 878.708 | .................................X | 878.708 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:38:12,477 | test | INFO | pool-17-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **......................................................................**..............................X | 2863.503 | | 752.016 | .........................*X | 752.016 | +---------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:38:15,490 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 11.34 2021-05-17 04:38:20,565 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 17.07 2021-05-17 04:38:25,634 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 23.17 2021-05-17 04:38:30,737 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 30.05 2021-05-17 04:38:35,901 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.39 2021-05-17 04:38:40,970 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 41.32 2021-05-17 04:38:46,092 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 52.43 2021-05-17 04:38:51,190 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 64.89 2021-05-17 04:38:56,288 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 68.01 2021-05-17 04:39:01,401 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.59 2021-05-17 04:39:06,506 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 78.17 2021-05-17 04:39:11,604 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 84.83 2021-05-17 04:39:16,674 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 91.09 2021-05-17 04:39:21,747 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: running, progress: 97.18 2021-05-17 04:39:31,846 | test | INFO | pool-17-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:39:31,882 | test | INFO | pool-17-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 119.616999865 sec 2021-05-17 04:39:31,931 | test | WARNING | Thread-50899 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:152G-58-736000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 04:39:31,931 | test | WARNING | Thread-50909 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:g%Wy1BIfdUC-58-736000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 04:39:31,933 | test | WARNING | Thread-50940 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:y67zkh1M-53-474000:oqENZ_TA-53-657000 create failed keys from task: [u'test_collections-5471', u'test_collections-5133', u'test_collections-5346', u'test_collections-5434'] 2021-05-17 04:39:31,931 | test | WARNING | Thread-50900 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:uOQIJLfZbdrf-58-736000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 04:39:31,944 | test | WARNING | Thread-50941 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:QKDoDq2U12XgBps5rfq-53-153000:52-53-287000 create failed keys from task: [u'test_collections-5532'] 2021-05-17 04:39:31,947 | test | WARNING | Thread-50942 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:QKDoDq2U12XgBps5rfq-53-153000:cjG9YkgZ5zCmfnB-53-288000 update failed keys from task: [u'test_collections-1926', u'test_collections-1743'] 2021-05-17 04:39:31,973 | test | WARNING | Thread-50937 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:v6Wo%-53-811000:9-53-911000 update failed keys from task: [u'test_collections-1667'] 2021-05-17 04:39:31,977 | test | WARNING | Thread-50937 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:v6Wo%-53-811000:9-53-911000 create failed keys from task: [u'test_collections-5148'] 2021-05-17 04:39:32,020 | test | WARNING | Thread-50976 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:0pmPSMvU8laL7TZ_-36-820000:VmKElTOdek_tdQ-36-966000 create failed keys from task: [u'test_collections-933', u'test_collections-3528', u'test_collections-3409', u'test_collections-2255', u'test_collections-3599'] 2021-05-17 04:39:32,028 | test | WARNING | Thread-50982 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:WH4KSU_DkilSdfD8-58-743000:DLD2JxlrsK_YhvXYv-38-796000 create failed keys from task: [u'test_collections-4837', u'test_collections-4652', u'test_collections-3961'] 2021-05-17 04:39:44,036 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:40:59,174 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 04:40:59,176 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 04:41:12,668 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 04:41:12,670 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 04:41:33,039 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 04:41:33,040 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 04:42:20,095 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-17 04:42:20,109 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 944.603s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 15 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_15 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_16 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_16', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 16, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 04:42:29,503 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #16 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:42:30,178 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #16 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:42:30,621 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8dc197 type=client open_count=1 channel=[id: 0xc2946aef, 0.0.0.0/0.0.0.0:51759] timeout=300.0>) 2021-05-17 04:42:51,372 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8dc198 type=client open_count=1 channel=[id: 0xbaa7c93f, 0.0.0.0/0.0.0.0:38476] timeout=300.0>) 2021-05-17 04:42:53,390 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8dc199 type=client open_count=1 channel=[id: 0x29a92465, 0.0.0.0/0.0.0.0:45833] timeout=300.0>) 2021-05-17 04:42:55,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8dc19a type=client open_count=1 channel=[id: 0xe247ce52, 0.0.0.0/0.0.0.0:50569] timeout=300.0>) 2021-05-17 04:43:19,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8dc19b type=client open_count=1 channel=[id: 0x30aa8757, 0.0.0.0/0.0.0.0:49732] timeout=300.0>) 2021-05-17 04:43:21,540 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8dc19c type=client open_count=1 channel=[id: 0xd3f012f9, 0.0.0.0/0.0.0.0:37915] timeout=300.0>) 2021-05-17 04:43:23,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8dc19d type=client open_count=1 channel=[id: 0xf5b51f1d, 0.0.0.0/0.0.0.0:36251] timeout=300.0>) 2021-05-17 04:43:47,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8dc19e type=client open_count=1 channel=[id: 0xd6a72ad8, 0.0.0.0/0.0.0.0:46785] timeout=300.0>) 2021-05-17 04:43:49,628 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8dc19f type=client open_count=1 channel=[id: 0x0939afe1, 0.0.0.0/0.0.0.0:35835] timeout=300.0>) 2021-05-17 04:43:51,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8dc1a0 type=client open_count=1 channel=[id: 0x9b605517, 0.0.0.0/0.0.0.0:37139] timeout=300.0>) 2021-05-17 04:44:15,749 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8dc1a1 type=client open_count=1 channel=[id: 0x17cc2f32, 0.0.0.0/0.0.0.0:36891] timeout=300.0>) 2021-05-17 04:44:17,766 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8dc1a2 type=client open_count=1 channel=[id: 0x4ec90674, 0.0.0.0/0.0.0.0:57704] timeout=300.0>) 2021-05-17 04:44:19,783 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8dc1a3 type=client open_count=1 channel=[id: 0xc1ceec4b, 0.0.0.0/0.0.0.0:59577] timeout=300.0>) 2021-05-17 04:44:43,793 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8dc1a4 type=client open_count=1 channel=[id: 0xd711ce17, 0.0.0.0/0.0.0.0:42069] timeout=300.0>) 2021-05-17 04:44:45,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8dc1a5 type=client open_count=1 channel=[id: 0x850eaa4d, 0.0.0.0/0.0.0.0:32851] timeout=300.0>) 2021-05-17 04:44:47,828 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 04:45:04,802 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 04:45:18,503 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 04:45:18,523 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #16 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:45:18,559 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #16 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:45:31,407 | test | INFO | pool-18-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 12.395 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------+--------------+ 2021-05-17 04:45:31,431 | test | INFO | pool-18-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:45:41,509 | test | INFO | pool-18-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:45:41,536 | test | INFO | pool-18-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1289999485 sec 2021-05-17 04:45:41,786 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 04:46:28,493 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 04:46:28,842 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 486633736 | 140804248 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 232756960 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 559738168 | 90042847 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 04:46:28,844 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:47:13,418 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:47:13,427 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:47:52,756 | test | INFO | pool-18-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......**..........................................................................................***...*X | 13628.168 | | 12560.240 | ...........................................................................................X | 12560.240 | +-----------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:47:54,151 | test | INFO | pool-18-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...........................................*X | 67.441 | | 7.493 | ...***.....**...........................................................................................*X | 154.181 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:47:55,664 | test | INFO | pool-18-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................**.........***.........**.............................................*.................*X | 123.527 | +-------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:47:55,931 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9731232467 | 6437847040 | 4293644288 | 2494160896 / 7511588864 | 82081 / 165494 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 100 | 6437847040 | 4206006272 | 2521268224 / 7511588864 | 81604 / 164436 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8428409015 | 6437847040 | 4247195648 | 2480582656 / 7511588864 | 80995 / 164593 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 100 | 6437847040 | 3809255424 | 3137015808 / 9530753024 | 82458 / 161323 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8702033481 | 6437847040 | 4216033280 | 2535878656 / 7511588864 | 82634 / 162716 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:48:09,519 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:49:36,588 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 163196376 | 264968642 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 246503040 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 591480040 | 406032268 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:49:36,588 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #16 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 04:49:46,788 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 04:49:46,789 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 04:49:46,789 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:50:26,427 | test | INFO | pool-18-thread-29 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 39.4560000896 seconds to complete 2021-05-17 04:50:26,431 | test | INFO | pool-18-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 04:52:31,493 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0569999217987 seconds 2021-05-17 04:52:31,493 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:52:31,493 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:52:38,769 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:53:05,114 | test | INFO | pool-18-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *................*.................................................**...................................*X | 8516.033 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:53:05,380 | test | INFO | pool-18-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 1443.055 | | 1398.090 | ...............................................................................................X | 1398.090 | | 1149.016 | ..............................................................................*X | 1149.016 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:53:05,420 | test | INFO | pool-18-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.............................**.......................................................................*X | 2587.585 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:53:05,680 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:53:05,733 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:53:18,796 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:53:19,092 | test | INFO | pool-18-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.526649111696 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 73.0665061691 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 77.8039294239 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 91.4568903349 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 74.304276481 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 04:53:19,197 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:53:24,282 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:53:25,766 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:53:29,512 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:53:34,674 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:53:40,194 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:53:45,426 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.56 2021-05-17 04:53:50,131 | test | INFO | pool-18-thread-17 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:Gz9HN8sevS-37-393000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:50,607 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 3.16 2021-05-17 04:53:51,142 | test | ERROR | pool-18-thread-7 | [task:call:3809] Failed to load 1 docs from 0 to 1000 2021-05-17 04:53:52,980 | test | INFO | pool-18-thread-27 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:XZSOAB-37-393000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:52,980 | test | ERROR | pool-18-thread-7 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 04:53:55,733 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 7.18 2021-05-17 04:53:56,352 | test | INFO | pool-18-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:s1sWK5-38-584000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5227 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,354 | test | INFO | pool-18-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:s1sWK5-38-584000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5227 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,651 | test | INFO | pool-18-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:s1sWK5-38-584000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5555 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,651 | test | INFO | pool-18-thread-6 | [table_view:display:72] Keys failed in bucket1:scope2:s1sWK5-38-584000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5555 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,713 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:XZSOAB-37-393000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5227 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,713 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:XZSOAB-37-393000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5227 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,990 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:XZSOAB-37-393000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5555 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:56,990 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:xD9jj-37-255000:XZSOAB-37-393000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5555 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:53:57,427 | test | ERROR | pool-18-thread-7 | [task:call:3809] Failed to load 2 docs from 5000 to 5800 2021-05-17 04:53:57,437 | test | ERROR | pool-18-thread-7 | [task:call:3809] Failed to load 2 docs from 5000 to 5800 2021-05-17 04:53:58,822 | test | INFO | pool-18-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***......................................................................**..............................*X | 8056.237 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:53:59,362 | test | INFO | pool-18-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 2235.068 | | 1807.618 | ...............................................................................X | 1807.618 | | 1624.285 | .......................................................................X | 1624.285 | | 780.374 | .................................X | 780.374 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:54:00,905 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 13.9 2021-05-17 04:54:01,237 | test | INFO | pool-18-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***..........................................**.........................................................*X | 2929.870 | | 511.808 | ................X | 511.808 | +---------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:54:06,016 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 20.93 2021-05-17 04:54:11,128 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 25.69 2021-05-17 04:54:16,217 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-17 04:54:21,286 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-17 04:54:26,371 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.93 2021-05-17 04:54:31,473 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 49.35 2021-05-17 04:54:36,563 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 59.48 2021-05-17 04:54:41,648 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: running, progress: 82.21 2021-05-17 04:54:51,763 | test | INFO | pool-18-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:54:51,801 | test | INFO | pool-18-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 92.7070000172 sec 2021-05-17 04:54:51,885 | test | WARNING | Thread-54937 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:xD9jj-37-255000:Gz9HN8sevS-37-393000 delete failed keys from task: [u'test_collections-96'] 2021-05-17 04:54:51,885 | test | WARNING | Thread-54936 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:xD9jj-37-255000:XZSOAB-37-393000 update failed keys from task: [u'test_collections-1926'] 2021-05-17 04:54:51,898 | test | WARNING | Thread-54936 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:xD9jj-37-255000:XZSOAB-37-393000 create failed keys from task: [u'test_collections-5227', u'test_collections-5555'] 2021-05-17 04:54:51,940 | test | WARNING | Thread-55016 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:s1sWK5-38-584000 create failed keys from task: [u'test_collections-5227', u'test_collections-5555'] 2021-05-17 04:55:04,242 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:56:22,032 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 04:56:22,032 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 04:56:37,217 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 04:56:37,219 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 04:57:01,891 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 04:57:01,891 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 04:57:50,992 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1270000935 seconds to complete 2021-05-17 04:57:51,007 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 930.874s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 16 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_16 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_17 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_17', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 17, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 04:58:00,482 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #17 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 04:58:01,161 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #17 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 04:58:01,667 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x979e0c type=client open_count=1 channel=[id: 0xd48e23a0, 0.0.0.0/0.0.0.0:41831] timeout=300.0>) 2021-05-17 04:58:22,467 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x979e0d type=client open_count=1 channel=[id: 0xb3b3c4db, 0.0.0.0/0.0.0.0:47338] timeout=300.0>) 2021-05-17 04:58:24,484 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x979e0e type=client open_count=1 channel=[id: 0x24eff0f2, 0.0.0.0/0.0.0.0:47750] timeout=300.0>) 2021-05-17 04:58:26,503 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x979e0f type=client open_count=1 channel=[id: 0xf58a2ca4, 0.0.0.0/0.0.0.0:41833] timeout=300.0>) 2021-05-17 04:58:50,447 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x979e10 type=client open_count=1 channel=[id: 0x8099deab, 0.0.0.0/0.0.0.0:36388] timeout=300.0>) 2021-05-17 04:58:52,466 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x979e11 type=client open_count=1 channel=[id: 0xee375a98, 0.0.0.0/0.0.0.0:55186] timeout=300.0>) 2021-05-17 04:58:54,483 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x979e12 type=client open_count=1 channel=[id: 0x3044646d, 0.0.0.0/0.0.0.0:43936] timeout=300.0>) 2021-05-17 04:59:18,638 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x979e13 type=client open_count=1 channel=[id: 0xa5f26527, 0.0.0.0/0.0.0.0:35394] timeout=300.0>) 2021-05-17 04:59:20,657 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x979e14 type=client open_count=1 channel=[id: 0x314596c3, 0.0.0.0/0.0.0.0:51579] timeout=300.0>) 2021-05-17 04:59:22,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x979e15 type=client open_count=1 channel=[id: 0x37ee58ce, 0.0.0.0/0.0.0.0:38279] timeout=300.0>) 2021-05-17 04:59:46,834 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x979e16 type=client open_count=1 channel=[id: 0xb4faad09, 0.0.0.0/0.0.0.0:37457] timeout=300.0>) 2021-05-17 04:59:48,851 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x979e17 type=client open_count=1 channel=[id: 0xbef19f54, 0.0.0.0/0.0.0.0:55900] timeout=300.0>) 2021-05-17 04:59:50,869 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x979e18 type=client open_count=1 channel=[id: 0x91ecadd7, 0.0.0.0/0.0.0.0:59668] timeout=300.0>) 2021-05-17 05:00:14,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x979e19 type=client open_count=1 channel=[id: 0xccc2ab4a, 0.0.0.0/0.0.0.0:59547] timeout=300.0>) 2021-05-17 05:00:16,931 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x979e1a type=client open_count=1 channel=[id: 0xe5b8a86b, 0.0.0.0/0.0.0.0:48669] timeout=300.0>) 2021-05-17 05:00:18,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 05:00:36,335 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 05:00:50,078 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 05:00:50,101 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #17 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:00:50,138 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #17 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:01:04,989 | test | INFO | pool-19-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 13.9064348928 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 05:01:05,013 | test | INFO | pool-19-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:01:15,089 | test | INFO | pool-19-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 05:01:15,118 | test | INFO | pool-19-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1289999485 sec 2021-05-17 05:01:15,368 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 05:01:59,663 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 05:02:00,036 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 590588696 | 113722248 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 222711088 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 640187784 | 71426654 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 05:02:00,036 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 05:02:45,595 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:02:45,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:03:27,506 | test | INFO | pool-19-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 164.205 | | 55.128 | ................................**.**X | 56.631 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 05:03:29,079 | test | INFO | pool-19-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***....................................................................................................*X | 97.843 | | 83.644 | ....................................................................................*X | 83.644 | | 69.438 | .....................................................................X | 69.438 | | 69.038 | .....................................................................*X | 69.038 | | 49.514 | .................................................X | 49.514 | +--------+-----------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 05:03:29,346 | test | INFO | pool-19-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***......................................................................................**...........*X | 12597.768 | | 12485.766 | ...............................................................................................X | 12485.766 | | 12429.324 | ...............................................................................................**....X | 12906.667 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 05:03:29,614 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.5593169413 | 6437847040 | 4305584128 | 2485514240 / 7511588864 | 100480 / 203071 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.4800086665 | 6437847040 | 4253851648 | 2476290048 / 7511588864 | 97178 / 198366 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9734161862 | 6437847040 | 4217348096 | 2518138880 / 7511588864 | 100943 / 201926 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.3221352739 | 6437847040 | 3844464640 | 3191840768 / 9530753024 | 73365 / 144770 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.5314680667 | 6437847040 | 4222402560 | 2512007168 / 7511588864 | 99687 / 195221 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 05:03:43,915 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 05:05:10,029 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 163391608 | 250714969 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 245759776 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 558602008 | 405666324 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 05:05:10,029 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #17 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:05:20,180 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 05:05:20,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 05:05:20,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 05:05:31,332 | test | INFO | pool-19-thread-25 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.86100006104 seconds to complete 2021-05-17 05:05:31,336 | test | INFO | pool-19-thread-25 | [common_lib:sleep:22] Sleep 8.86100006104 seconds. Reason: Wait after rebalance complete 2021-05-17 05:07:40,257 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 05:07:40,259 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 05:07:40,259 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:07:48,183 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:08:18,717 | test | INFO | pool-19-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 2070.668 | | 1501.555 | .......................................................................**X | 1501.555 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:08:19,403 | test | INFO | pool-19-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 2169.405 | | 904.411 | ........................................**X | 904.411 | | 51.002 | .X | 51.002 | +---------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:08:19,973 | test | INFO | pool-19-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.............................................**.......................................................**X | 14238.722 | | 13185.504 | ...........................................................................................X | 13185.504 | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 05:08:20,450 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 05:08:20,522 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 05:08:32,270 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:08:32,536 | test | INFO | pool-19-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.698379891993 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 87.5478191143 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 87.5857579248 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 48.1525307912 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 89.4102525581 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 05:08:32,608 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:08:37,717 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:08:38,953 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:08:45,482 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:08:47,676 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:47,677 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,342 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:cyJ76NfjZk-dTIrs-37-839000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,344 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:cyJ76NfjZk-dTIrs-37-839000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,348 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:DlGg-37-531000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2827 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,349 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:DlGg-37-531000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2827 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,618 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,618 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,825 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:DWm4VHr5eorX2ihq-46-426000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1750 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,924 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3961 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:48,924 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3961 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,138 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,140 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-526 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,233 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:DlGg-37-531000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3393 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3203 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,233 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:DlGg-37-531000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3393 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3203 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,239 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:cyJ76NfjZk-dTIrs-37-839000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3393 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3203 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,240 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:cyJ76NfjZk-dTIrs-37-839000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3393 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3203 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,255 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:DWm4VHr5eorX2ihq-46-426000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1926 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1929 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,381 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4194 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4004 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:49,381 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4194 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4004 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,075 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,075 | test | INFO | pool-19-thread-7 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,382 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:8a9CpVSvR6gJj2%-37-233000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,384 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket1:vR-jVpW-37-124000:8a9CpVSvR6gJj2%-37-233000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,407 | test | INFO | pool-19-thread-9 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:Tr9DxL_ppR7BK-37-531000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,408 | test | INFO | pool-19-thread-9 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:Tr9DxL_ppR7BK-37-531000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,756 | test | INFO | pool-19-thread-9 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:Tr9DxL_ppR7BK-37-531000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4733 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,756 | test | INFO | pool-19-thread-9 | [table_view:display:72] Keys failed in bucket1:HjB0fg-37-365000:Tr9DxL_ppR7BK-37-531000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4733 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:50,878 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.65 2021-05-17 05:08:51,325 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 05:08:51,326 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 05:08:52,029 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 05:08:52,033 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 05:08:52,108 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:52,109 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1217 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1387 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:52,328 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 05:08:52,408 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:52,410 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1565 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,029 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2042 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,029 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2042 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,368 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,368 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,454 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:tvsAjQVpy-47-35000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-439 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:53,618 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:tvsAjQVpy-47-35000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:54,160 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2909 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2899 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:54,160 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2909 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2899 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,418 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 05:08:55,438 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4268 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,440 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4268 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,658 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4578 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,658 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4578 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,956 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:55,957 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-251 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:56,118 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4974 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:56,118 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:wJqJp-37-714000:As-37-838000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4974 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:56,119 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 16 docs from 0 to 5000 2021-05-17 05:08:56,253 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 2.64 2021-05-17 05:08:56,338 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:DWm4VHr5eorX2ihq-46-426000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-45 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:57,444 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:57,446 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1879 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:58,404 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2986 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:58,405 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2816 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2986 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:58,625 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3048 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:58,625 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:M-15-436000:iVCRUYoN-38-793000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3048 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:08:59,428 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1106 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1096 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1121 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:01,447 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 4.33 2021-05-17 05:09:01,858 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 05:09:01,875 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 05:09:01,882 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 1 docs from 0 to 1000 2021-05-17 05:09:01,894 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 05:09:01,974 | test | INFO | pool-19-thread-10 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-199 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,109 | test | INFO | pool-19-thread-10 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,303 | test | INFO | pool-19-thread-10 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-402 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,461 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,509 | test | INFO | pool-19-thread-10 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-796 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,709 | test | INFO | pool-19-thread-10 | [table_view:display:72] Keys failed in bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,869 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:zYYbqL2w8N%_UA-46-427000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:02,871 | test | INFO | pool-19-thread-11 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1458 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:03,243 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 8 docs from 0 to 1000 2021-05-17 05:09:03,243 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 05:09:03,246 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 05:09:06,641 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 6.02 2021-05-17 05:09:08,655 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:08,875 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:09,196 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1839 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:09,197 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 05:09:10,470 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:zYYbqL2w8N%_UA-46-427000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-207 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-397 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:10,486 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-207 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-397 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:10,611 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in bucket1:jJtDMp5gdm9Rgwk-46-273000:zYYbqL2w8N%_UA-46-427000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-575 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:10,631 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-575 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:11,986 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 8.11 2021-05-17 05:09:12,456 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 05:09:12,457 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 05:09:17,161 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 10.04 2021-05-17 05:09:17,220 | test | INFO | pool-19-thread-22 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:18,990 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 1 docs from 0 to 1000 2021-05-17 05:09:19,586 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5151 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:19,588 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5151 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:09:20,132 | test | ERROR | pool-19-thread-20 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 05:09:22,132 | test | INFO | pool-19-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.................................................*...............*....................................*X | 2896.212 | | 199.931 | .....*X | 199.931 | | 94.100 | ..X | 94.100 | +---------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:09:22,237 | test | INFO | pool-19-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *........................................*..............*..................................*............X | 2475.051 | | 1287.544 | ...................................................*X | 1287.544 | | 482.662 | ..................X | 482.662 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:09:22,342 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 12.77 2021-05-17 05:09:22,674 | test | INFO | pool-19-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +----------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...............*.....*............................................................**....................*X | 10355.970 | | 9244.600 | ........................................................................................X | 9244.600 | +----------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 05:09:27,469 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 16.19 2021-05-17 05:09:32,556 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 21.31 2021-05-17 05:09:37,661 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 27.3 2021-05-17 05:09:42,727 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 05:09:47,792 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 36.16 2021-05-17 05:09:52,881 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 46.77 2021-05-17 05:09:58,003 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 58.23 2021-05-17 05:10:03,115 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 05:10:08,216 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 69.53 2021-05-17 05:10:13,282 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 74.78 2021-05-17 05:10:18,362 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 80.07 2021-05-17 05:10:23,430 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 85.95 2021-05-17 05:10:28,503 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 91.67 2021-05-17 05:10:33,611 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: running, progress: 97.03 2021-05-17 05:10:43,713 | test | INFO | pool-19-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 05:10:43,750 | test | INFO | pool-19-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 131.213999987 sec 2021-05-17 05:10:43,786 | test | WARNING | Thread-58554 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 update failed keys from task: [u'test_collections-1106', u'test_collections-1096', u'test_collections-1121'] 2021-05-17 05:10:43,786 | test | WARNING | Thread-58553 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aSXJt-46-899000:tvsAjQVpy-47-35000 delete failed keys from task: [u'test_collections-439', u'test_collections-610', u'test_collections-780'] 2021-05-17 05:10:43,786 | test | WARNING | Thread-58552 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:vR-jVpW-37-124000:8a9CpVSvR6gJj2%-37-233000 create failed keys from task: [u'test_collections-4428'] 2021-05-17 05:10:43,786 | test | WARNING | Thread-58555 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HjB0fg-37-365000:DlGg-37-531000 create failed keys from task: [u'test_collections-3393', u'test_collections-2827', u'test_collections-3203'] 2021-05-17 05:10:43,801 | test | WARNING | Thread-58596 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 update failed keys from task: [u'test_collections-1458', u'test_collections-1164'] 2021-05-17 05:10:43,801 | test | WARNING | Thread-58578 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:M-15-436000:iVCRUYoN-38-793000 create failed keys from task: [u'test_collections-2816', u'test_collections-2986', u'test_collections-1879', u'test_collections-251', u'test_collections-3048'] 2021-05-17 05:10:43,801 | test | WARNING | Thread-58551 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:vR-jVpW-37-124000:DBxPtp7T-37-232000 create failed keys from task: [u'test_collections-4194', u'test_collections-4428', u'test_collections-3246', u'test_collections-3961', u'test_collections-4004'] 2021-05-17 05:10:43,801 | test | WARNING | Thread-58556 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HjB0fg-37-365000:Tr9DxL_ppR7BK-37-531000 create failed keys from task: [u'test_collections-4428', u'test_collections-4733'] 2021-05-17 05:10:43,808 | test | WARNING | Thread-58595 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 update failed keys from task: [u'test_collections-1839', u'test_collections-1634', u'test_collections-1429'] 2021-05-17 05:10:43,815 | test | WARNING | Thread-58554 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aSXJt-46-899000:YIUvsQOIKbFCRVdJq-47-34000 delete failed keys from task: [u'test_collections-606', u'test_collections-535', u'test_collections-402', u'test_collections-960', u'test_collections-796', u'test_collections-199', u'test_collections-592', u'test_collections-370'] 2021-05-17 05:10:43,816 | test | WARNING | Thread-58596 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PRmiJSOE5gna97y-46-593000:RE-46-717000 delete failed keys from task: [u'test_collections-207', u'test_collections-575', u'test_collections-397'] 2021-05-17 05:10:43,826 | test | WARNING | Thread-58595 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 delete failed keys from task: [u'test_collections-13'] 2021-05-17 05:10:43,836 | test | WARNING | Thread-58594 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:jJtDMp5gdm9Rgwk-46-273000:DWm4VHr5eorX2ihq-46-426000 update failed keys from task: [u'test_collections-1926', u'test_collections-1929', u'test_collections-1750'] 2021-05-17 05:10:43,841 | test | WARNING | Thread-58595 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:PRmiJSOE5gna97y-46-593000:MpMyVNWCd-46-716000 create failed keys from task: [u'test_collections-5151'] 2021-05-17 05:10:43,865 | test | WARNING | Thread-58594 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:jJtDMp5gdm9Rgwk-46-273000:DWm4VHr5eorX2ihq-46-426000 delete failed keys from task: [u'test_collections-45'] 2021-05-17 05:10:43,904 | test | WARNING | Thread-58593 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:jJtDMp5gdm9Rgwk-46-273000:zYYbqL2w8N%_UA-46-427000 update failed keys from task: [u'test_collections-1786', u'test_collections-1616'] 2021-05-17 05:10:43,905 | test | WARNING | Thread-58631 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wJqJp-37-714000:As-37-838000 create failed keys from task: [u'test_collections-2519', u'test_collections-526', u'test_collections-1565', u'test_collections-2899', u'test_collections-363', u'test_collections-581', u'test_collections-4578', u'test_collections-4974', u'test_collections-1387', u'test_collections-2909', u'test_collections-411', u'test_collections-2489', u'test_collections-1217', u'test_collections-1339', u'test_collections-2042', u'test_collections-4268'] 2021-05-17 05:10:43,914 | test | WARNING | Thread-58630 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wJqJp-37-714000:cyJ76NfjZk-dTIrs-37-839000 create failed keys from task: [u'test_collections-3393', u'test_collections-2827', u'test_collections-3203'] 2021-05-17 05:10:43,917 | test | WARNING | Thread-58593 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:jJtDMp5gdm9Rgwk-46-273000:zYYbqL2w8N%_UA-46-427000 delete failed keys from task: [u'test_collections-207', u'test_collections-575', u'test_collections-397'] 2021-05-17 05:10:55,974 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 05:12:12,015 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 05:12:12,015 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 05:12:21,788 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 05:12:21,789 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 05:12:40,200 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 05:12:40,200 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 05:13:34,276 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1229999065 seconds to complete 2021-05-17 05:13:34,293 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 943.254s During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 17 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_17 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_18 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'recovery_type': 'delta', 'step_count': '1', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_18', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 18, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 05:13:43,796 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #18 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:13:44,486 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #18 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:13:44,903 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xa132f0 type=client open_count=1 channel=[id: 0x975bb5fa, 0.0.0.0/0.0.0.0:53394] timeout=300.0>) 2021-05-17 05:14:05,614 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xa132f1 type=client open_count=1 channel=[id: 0xf79aea51, 0.0.0.0/0.0.0.0:45184] timeout=300.0>) 2021-05-17 05:14:07,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xa132f2 type=client open_count=1 channel=[id: 0xb8a2bbab, 0.0.0.0/0.0.0.0:56300] timeout=300.0>) 2021-05-17 05:14:09,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xa132f3 type=client open_count=1 channel=[id: 0x6800be68, 0.0.0.0/0.0.0.0:49503] timeout=300.0>) 2021-05-17 05:14:11,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xa132f4 type=client open_count=1 channel=[id: 0x79708b58, 0.0.0.0/0.0.0.0:36585] timeout=300.0>) 2021-05-17 05:14:13,688 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xa132f5 type=client open_count=1 channel=[id: 0x11eabf17, 0.0.0.0/0.0.0.0:60497] timeout=300.0>) 2021-05-17 05:14:39,209 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xa132f6 type=client open_count=1 channel=[id: 0x27d2b6db, 0.0.0.0/0.0.0.0:33489] timeout=300.0>) 2021-05-17 05:14:41,227 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xa132f7 type=client open_count=1 channel=[id: 0x62593b12, 0.0.0.0/0.0.0.0:41468] timeout=300.0>) 2021-05-17 05:14:43,244 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xa132f8 type=client open_count=1 channel=[id: 0x8e839b72, 0.0.0.0/0.0.0.0:40245] timeout=300.0>) 2021-05-17 05:15:07,378 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xa132f9 type=client open_count=1 channel=[id: 0x1a887a4b, 0.0.0.0/0.0.0.0:44473] timeout=300.0>) 2021-05-17 05:15:09,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xa132fa type=client open_count=1 channel=[id: 0xf87b859e, 0.0.0.0/0.0.0.0:37372] timeout=300.0>) 2021-05-17 05:15:11,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xa132fb type=client open_count=1 channel=[id: 0x69e93964, 0.0.0.0/0.0.0.0:50828] timeout=300.0>) 2021-05-17 05:15:35,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xa132fc type=client open_count=1 channel=[id: 0x906b1007, 0.0.0.0/0.0.0.0:60518] timeout=300.0>) 2021-05-17 05:15:37,788 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xa132fd type=client open_count=1 channel=[id: 0xb6749536, 0.0.0.0/0.0.0.0:53783] timeout=300.0>) 2021-05-17 05:15:39,805 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xa132fe type=client open_count=1 channel=[id: 0x82f6fdd7, 0.0.0.0/0.0.0.0:56561] timeout=300.0>) 2021-05-17 05:16:03,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xa132ff type=client open_count=1 channel=[id: 0x2ec7f5a2, 0.0.0.0/0.0.0.0:38725] timeout=300.0>) 2021-05-17 05:16:05,832 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xa13300 type=client open_count=1 channel=[id: 0x7a549453, 0.0.0.0/0.0.0.0:38873] timeout=300.0>) 2021-05-17 05:16:07,849 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 05:16:24,977 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 05:16:38,743 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 05:16:38,765 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #18 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:16:38,802 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #18 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:16:52,259 | test | INFO | pool-20-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.49490841819 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 05:16:52,292 | test | INFO | pool-20-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:17:02,368 | test | INFO | pool-20-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 05:17:02,394 | test | INFO | pool-20-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1349999905 sec 2021-05-17 05:17:02,651 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 05:17:54,697 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 05:17:54,973 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 452824248 | 155151230 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 214169424 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 465939720 | 133905578 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 05:17:54,973 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 05:18:38,384 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:18:38,391 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:19:19,704 | test | INFO | pool-20-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 119.839 | | 53.411 | ...........................................*X | 53.411 | | 38.143 | ..............................**.....................................X | 82.669 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 05:19:21,450 | test | INFO | pool-20-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.........................................................***.......................................**....**X | 13376.286 | | 12487.412 | ............................................................................................X | 12487.412 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 05:19:22,892 | test | INFO | pool-20-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..........................................................................................*X | 93.752 | | 49.481 | ..............................................X | 49.481 | | 33.406 | ...............................*..*...........................**.......................................*X | 103.537 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 05:19:23,234 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 100 | 6437847040 | 4295577600 | 2482274304 / 7511588864 | 94345 / 187491 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 93.4324659232 | 6437847040 | 4216143872 | 2517794816 / 7511588864 | 92947 / 186658 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 95.9837078089 | 6437847040 | 4207263744 | 2514165760 / 7511588864 | 93319 / 186848 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.975 | 6437847040 | 3850063872 | 3128225792 / 9530753024 | 93545 / 186209 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 92.6971356079 | 6437847040 | 4234133504 | 2506174464 / 7511588864 | 93506 / 186551 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 05:19:36,953 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 05:21:02,720 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162320440 | 265917656 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 244916896 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 581772808 | 413594805 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 05:21:02,721 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #18 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 05:21:13,009 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 05:21:13,009 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 05:21:13,009 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:765] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 05:21:24,051 | test | INFO | pool-20-thread-4 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.28700017929 seconds to complete 2021-05-17 05:21:24,056 | test | INFO | pool-20-thread-4 | [common_lib:sleep:22] Sleep 8.28700017929 seconds. Reason: Wait after rebalance complete 2021-05-17 05:23:32,405 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0579998493195 seconds 2021-05-17 05:23:32,407 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 05:23:32,407 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:23:40,309 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:24:13,354 | test | INFO | pool-20-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..........................*..................*........................................................*X | 2288.332 | | 193.801 | .......*X | 193.801 | +---------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:24:14,908 | test | INFO | pool-20-thread-30 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **............................**.......................................................................**.X | 13386.455 | | 13336.906 | ..................................................................................................*X | 13336.906 | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 05:24:15,434 | test | INFO | pool-20-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *................................*.....................................*.........*......................*X | 1805.757 | | 42.542 | .**X | 42.542 | +--------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:24:15,707 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 05:24:15,766 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 05:24:27,443 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:24:27,693 | test | INFO | pool-20-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.545018167272 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 65.3639106015 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 57.4717448004 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 80.3451630216 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 58.6714818344 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 05:24:27,766 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:24:32,987 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:24:33,931 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:24:38,242 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:24:44,760 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:24:53,098 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:24:55,960 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1110 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:24:56,581 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1662 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:24:56,880 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1901 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1891 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:24:58,345 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 2.52 2021-05-17 05:24:58,674 | test | ERROR | pool-20-thread-15 | [task:call:3809] Failed to load 5 docs from 1000 to 1960 2021-05-17 05:24:59,346 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in bucket1:9kJD2rf0m06Q71R9tA-32-228000:2_05iYdZB_y1Kf6l-32-396000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:24:59,346 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in bucket1:9kJD2rf0m06Q71R9tA-32-228000:2_05iYdZB_y1Kf6l-32-396000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:03,516 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 7.52 2021-05-17 05:25:04,349 | test | ERROR | pool-20-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 05:25:05,921 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in bucket1:0WAi10q-37-616000:E_m7OLdLo3-37-722000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5222 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5227 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:05,923 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in bucket1:0WAi10q-37-616000:E_m7OLdLo3-37-722000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5222 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5227 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:07,032 | test | ERROR | pool-20-thread-15 | [task:call:3809] Failed to load 2 docs from 5000 to 5800 2021-05-17 05:25:07,884 | test | INFO | pool-20-thread-7 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5148 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:07,884 | test | INFO | pool-20-thread-7 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5148 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:08,766 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 13.11 2021-05-17 05:25:10,453 | test | ERROR | pool-20-thread-15 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 05:25:12,440 | test | INFO | pool-20-thread-21 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:m-37-398000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5549 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:12,440 | test | INFO | pool-20-thread-21 | [table_view:display:72] Keys failed in bucket1:raHm%0-37-200000:m-37-398000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5549 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 05:25:14,032 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 18.86 2021-05-17 05:25:14,819 | test | ERROR | pool-20-thread-15 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 05:25:18,112 | test | INFO | pool-20-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **......................................................................*..............................*X | 3100.524 | | 1144.718 | ...................................*X | 1144.718 | | 739.762 | ......................*X | 739.762 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:25:18,214 | test | INFO | pool-20-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **................................................*...................................................*X | 8627.102 | | 7751.440 | ........................................................................................**..........*.X | 8688.492 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:25:18,332 | test | INFO | pool-20-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **........................*............................................................................*X | 2894.281 | | 1012.157 | .................................**.......X | 1208.481 | | 550.694 | ..................X | 550.694 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 05:25:19,253 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 22.75 2021-05-17 05:25:24,322 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 24.84 2021-05-17 05:25:29,391 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 29.24 2021-05-17 05:25:34,500 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-17 05:25:39,628 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 37.88 2021-05-17 05:25:44,776 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 46.7 2021-05-17 05:25:49,884 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 56.33 2021-05-17 05:25:54,964 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.24 2021-05-17 05:26:00,075 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 83.06 2021-05-17 05:26:05,140 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: running, progress: 88.13 2021-05-17 05:26:15,236 | test | INFO | pool-20-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 05:26:15,273 | test | INFO | pool-20-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 107.58100009 sec 2021-05-17 05:26:15,372 | test | WARNING | Thread-62625 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:9kJD2rf0m06Q71R9tA-32-228000:2_05iYdZB_y1Kf6l-32-396000 create failed keys from task: [u'test_collections-1379'] 2021-05-17 05:26:15,380 | test | WARNING | Thread-62659 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:0WAi10q-37-616000:E_m7OLdLo3-37-722000 create failed keys from task: [u'test_collections-5222', u'test_collections-5227'] 2021-05-17 05:26:15,414 | test | WARNING | Thread-62736 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:raHm%0-37-200000:m-37-398000 create failed keys from task: [u'test_collections-5549'] 2021-05-17 05:26:15,414 | test | WARNING | Thread-62735 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 update failed keys from task: [u'test_collections-1080', u'test_collections-1901', u'test_collections-1110', u'test_collections-1891', u'test_collections-1662'] 2021-05-17 05:26:15,434 | test | WARNING | Thread-62735 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:raHm%0-37-200000:ql79TD%hNQ-37-397000 create failed keys from task: [u'test_collections-5148'] 2021-05-17 05:26:27,490 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 05:27:45,736 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 05:27:45,737 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 05:27:58,247 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 05:27:58,249 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 05:28:16,710 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 05:28:16,711 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 05:29:09,904 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.126999855 seconds to complete 2021-05-17 05:29:09,921 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 935.633s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 18 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_18 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_19 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_19', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 19, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 05:29:19,513 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 05:29:20,209 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 05:29:20,717 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xab29ad type=client open_count=1 channel=[id: 0x680b6efa, 0.0.0.0/0.0.0.0:57944] timeout=300.0>) 2021-05-17 05:29:41,480 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xab29ae type=client open_count=1 channel=[id: 0xfc9818f1, 0.0.0.0/0.0.0.0:56826] timeout=300.0>) 2021-05-17 05:29:43,497 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xab29af type=client open_count=1 channel=[id: 0xe291a57c, 0.0.0.0/0.0.0.0:55254] timeout=300.0>) 2021-05-17 05:29:45,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xab29b0 type=client open_count=1 channel=[id: 0xa185d95c, 0.0.0.0/0.0.0.0:35472] timeout=300.0>) 2021-05-17 05:30:09,553 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xab29b1 type=client open_count=1 channel=[id: 0xa0acec7a, 0.0.0.0/0.0.0.0:56896] timeout=300.0>) 2021-05-17 05:30:11,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xab29b2 type=client open_count=1 channel=[id: 0x3c949baa, 0.0.0.0/0.0.0.0:59058] timeout=300.0>) 2021-05-17 05:30:13,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xab29b3 type=client open_count=1 channel=[id: 0x44bd219e, 0.0.0.0/0.0.0.0:48519] timeout=300.0>) 2021-05-17 05:30:37,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xab29b4 type=client open_count=1 channel=[id: 0xd8890a29, 0.0.0.0/0.0.0.0:54487] timeout=300.0>) 2021-05-17 05:30:39,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xab29b5 type=client open_count=1 channel=[id: 0xab1bf264, 0.0.0.0/0.0.0.0:32971] timeout=300.0>) 2021-05-17 05:30:41,835 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xab29b6 type=client open_count=1 channel=[id: 0xe20bd1c2, 0.0.0.0/0.0.0.0:46471] timeout=300.0>) 2021-05-17 05:31:05,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xab29b7 type=client open_count=1 channel=[id: 0xd7131666, 0.0.0.0/0.0.0.0:55060] timeout=300.0>) 2021-05-17 05:31:07,964 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xab29b8 type=client open_count=1 channel=[id: 0x8c8d32ba, 0.0.0.0/0.0.0.0:42769] timeout=300.0>) 2021-05-17 05:31:09,984 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xab29b9 type=client open_count=1 channel=[id: 0xbdf6f608, 0.0.0.0/0.0.0.0:35343] timeout=300.0>) 2021-05-17 05:31:33,926 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xab29ba type=client open_count=1 channel=[id: 0x35b7a1db, 0.0.0.0/0.0.0.0:39441] timeout=300.0>) 2021-05-17 05:31:35,943 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xab29bb type=client open_count=1 channel=[id: 0x1f764bfe, 0.0.0.0/0.0.0.0:38456] timeout=300.0>) 2021-05-17 05:31:37,961 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 05:31:55,042 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 05:32:08,720 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 05:32:08,743 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 05:32:08,779 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 05:32:22,276 | test | INFO | pool-21-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.12666666667 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 05:32:22,299 | test | INFO | pool-21-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 05:32:32,372 | test | INFO | pool-21-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 05:32:32,414 | test | INFO | pool-21-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.138999939 sec 2021-05-17 05:32:32,680 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 05:33:28,369 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 05:33:28,676 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 613458552 | 107095004 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 559841296 | 125200044 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 635706856 | 89597878 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 05:33:28,677 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 05:34:12,015 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:34:12,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 05:52:49,315 | test | INFO | pool-21-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **.............................*..****X | 9.005 | | 2.399 | .......X | 2.399 | | 0.000 | ***********************************************************************************************************************************.....****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | **********************************..........*****X | 3.000 | | 0.000 | ***************************************.....*****...*.......****X | 4.401 | | 0.200 | X | 0.200 | | 0.100 | *****..........X | 2.900 | | 2.900 | .........***X | 2.900 | | 2.100 | ......X | 2.100 | | 0.100 | X | 0.100 | | 0.000 | **********....*.***X | 1.500 | | 1.100 | ..X | 1.100 | | 0.100 | X | 0.100 | | 0.000 | *******************...*****.X | 1.200 | | 0.400 | *****....****X | 1.500 | | 0.700 | .X | 0.700 | | 0.100 | X | 0.100 | | 0.000 | *********************..*...****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | **********************...*.****X | 1.400 | | 1.000 | ..X | 1.000 | | 0.000 | *****....*.***X | 1.500 | | 1.100 | ..X | 1.100 | | 0.000 | *****************************...******..............****X | 4.900 | | 2.001 | ......*****...........*.***X | 5.500 | | 3.900 | ............X | 3.900 | | 0.000 | *****........*.......****X | 4.500 | | 0.000 | *****..*..*.****.....X | 3.001 | | 2.901 | .........****X | 2.901 | | 0.100 | X | 0.100 | | 0.000 | ****.........*...........................*****X | 10.500 | | 1.500 | ....*****....................................................*....****...............*............****X | 25.200 | | 17.997 | ..............................................................X | 17.997 | | 17.197 | ...........................................................****X | 17.197 | | 2.400 | .......X | 2.400 | | 1.500 | ....*****.............................*.............***X | 13.399 | | 13.397 | ..............................................X | 13.397 | | 8.201 | ...........................X | 8.201 | | 7.701 | ..........................****.............*...........................................................****X | 28.496 | | 19.700 | ....................................................................X | 19.700 | | 14.599 | ..................................................*****.............****X | 18.306 | | 13.104 | ............................................X | 13.104 | | 0.000 | *****************.......................*.****X | 7.101 | | 5.799 | ...................******...****X | 6.600 | | 4.400 | ..............X | 4.400 | | 0.400 | X | 0.400 | | 0.100 | ****...............................*..........................*...***X | 17.305 | | 11.500 | .......................................X | 11.500 | | 11.100 | .....................................*****.....*..***X | 13.098 | | 10.698 | ....................................X | 10.698 | | 2.753 | ........X | 2.753 | | 1.953 | .....*****....*****......*..............*.****................*.....****X | 15.000 | | 8.000 | ...........................X | 8.000 | | 0.000 | **********************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 05:52:50,000 | test | INFO | pool-21-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............................*........****.............*................................................****X | 896.618 | | 777.403 | .....................................................................................X | 777.403 | | 715.246 | ..............................................................................****X | 715.246 | | 592.894 | .................................................................X | 592.894 | | 590.747 | ................................................................****X | 590.747 | | 566.956 | ..............................................................****X | 566.956 | | 562.987 | .............................................................X | 562.987 | | 513.483 | ........................................................*****....*......****X | 604.978 | | 579.206 | ...............................................................X | 579.206 | | 546.781 | ...........................................................***X | 546.781 | | 524.995 | .........................................................*X | 524.995 | | 505.956 | .......................................................***X | 505.956 | | 447.093 | ................................................*.*****..X | 468.553 | | 454.477 | .................................................*****...*****...X | 508.256 | | 488.094 | .....................................................****X | 488.094 | | 455.909 | .................................................*..****....X | 505.248 | | 498.386 | ......................................................****X | 498.386 | | 488.109 | .....................................................X | 488.109 | | 483.694 | ....................................................****X | 483.694 | | 452.670 | .................................................X | 452.670 | | 437.209 | ...............................................*****.****X | 441.753 | | 414.017 | .............................................X | 414.017 | | 409.212 | ............................................*****....*.*****....X | 489.160 | | 486.043 | .....................................................*****........*......***X | 614.552 | | 564.898 | ..............................................................X | 564.898 | | 532.780 | ..........................................................****X | 532.780 | | 446.742 | ................................................X | 446.742 | | 433.070 | ...............................................****X | 433.070 | | 426.774 | ..............................................*****...X | 450.906 | | 448.324 | .................................................****X | 448.324 | | 369.905 | ........................................*****...X | 397.753 | | 391.564 | ..........................................*****..*...***X | 436.503 | | 430.903 | ...............................................*.****X | 440.610 | | 401.466 | ...........................................*..*****..*****X | 432.635 | | 432.190 | ...............................................X | 432.190 | | 429.100 | ..............................................***X | 429.100 | | 413.970 | .............................................*..*****X | 430.909 | | 420.759 | .............................................*****..X | 433.057 | | 427.559 | ..............................................*****.....*..****X | 491.637 | | 440.825 | ................................................*****.X | 449.811 | | 447.912 | ................................................****X | 447.912 | | 422.196 | ..............................................******.*****.X | 445.157 | | 442.266 | ................................................***X | 442.266 | | 428.459 | ..............................................X | 428.459 | | 416.808 | .............................................*****.*.*****.*.****X | 453.861 | | 432.020 | ...............................................X | 432.020 | | 424.558 | ..............................................****.*****....*****X | 471.009 | | 427.414 | ..............................................*****.X | 432.051 | | 425.962 | ..............................................*****..*****.....*....**X | 520.410 | | 518.590 | ........................................................X | 518.590 | | 472.980 | ...................................................*****..X | 485.386 | | 472.739 | ...................................................*****X | 472.739 | | 461.353 | ..................................................****X | 461.353 | | 450.210 | .................................................X | 450.210 | | 432.626 | ...............................................***X | 432.626 | | 418.515 | .............................................*.*****.*....****X | 471.919 | | 458.407 | ..................................................X | 458.407 | | 447.218 | ................................................*****..*.***X | 469.673 | | 453.969 | .................................................*X | 454.706 | | 409.263 | ............................................***X | 409.263 | | 374.169 | ........................................****X | 374.169 | | 368.684 | ........................................*..X | 391.721 | | 378.816 | .........................................***X | 378.816 | | 370.302 | ........................................X | 370.302 | | 364.785 | .......................................*****........*.****X | 445.879 | | 443.576 | ................................................*.*****.......X | 518.382 | | 512.483 | ........................................................****X | 512.483 | | 444.457 | ................................................*.***X | 452.754 | | 425.220 | ..............................................X | 425.220 | | 419.528 | .............................................X | 419.528 | | 399.534 | ...........................................****..*..****X | 437.153 | | 403.554 | ............................................X | 403.554 | | 393.144 | ..........................................****X | 393.144 | | 367.239 | .......................................X | 367.239 | | 359.532 | .......................................****.*....****X | 411.425 | | 409.297 | ............................................*****.*...****X | 448.143 | | 437.702 | ...............................................X | 437.702 | | 357.278 | ......................................*****...........****X | 456.507 | | 442.575 | ................................................X | 442.575 | | 408.304 | ............................................*****.....*.****X | 465.264 | | 438.480 | ...............................................X | 438.480 | | 392.132 | ..........................................****...*....**********X | 452.628 | | 421.501 | ..............................................X | 421.501 | | 408.476 | ............................................***X | 408.476 | | 362.311 | .......................................X | 362.311 | | 342.709 | .....................................*****...........****X | 442.719 | | 422.729 | ..............................................X | 422.729 | | 419.082 | .............................................X | 419.082 | | 415.882 | .............................................***X | 415.882 | | 370.859 | ........................................*****X | 374.759 | | 371.849 | ........................................X | 371.849 | | 351.251 | ......................................***X | 351.251 | | 330.525 | ...................................X | 330.525 | | 314.267 | ..................................X | 314.267 | | 300.459 | ................................****...*.....*****...X | 403.187 | | 391.696 | ..........................................*****......*****X | 443.071 | | 433.033 | ...............................................****X | 433.033 | | 431.932 | ...............................................X | 431.932 | | 390.222 | ..........................................****X | 390.222 | | 378.397 | .........................................*..****X | 395.850 | | 393.171 | ..........................................X | 393.171 | | 378.554 | .........................................******.....****X | 428.125 | | 411.001 | ............................................****X | 411.001 | | 403.516 | ............................................X | 403.516 | | 376.089 | ........................................*****..*.........****X | 468.701 | | 401.578 | ...........................................X | 401.578 | | 346.553 | .....................................*****.......*...*****.****X | 439.953 | | 424.447 | ..............................................X | 424.447 | | 378.229 | .........................................******...*****..........*.***X | 506.108 | | 495.586 | ......................................................X | 495.586 | | 468.023 | ...................................................*****X | 468.023 | | 450.642 | .................................................*****..X | 473.334 | | 469.015 | ...................................................*****..*****.*....****X | 533.276 | | 506.266 | .......................................................X | 506.266 | | 364.338 | .......................................*X | 364.338 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 05:52:50,897 | test | INFO | pool-21-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***************************************************************************************************************************************************************************************************************************************.......****X | 3.000 | | 2.200 | ....X | 2.200 | | 0.000 | *****.*..****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | ****************************************************.......*****...*.....****X | 6.001 | | 4.400 | ..........X | 4.400 | | 0.000 | *****...*****X | 1.500 | | 0.000 | *****...*........****X | 4.500 | | 2.100 | ....X | 2.100 | | 0.300 | X | 0.300 | | 0.000 | ****..*.****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | *****************...........******...****X | 5.400 | | 3.400 | .......X | 3.400 | | 0.600 | ****X | 0.600 | | 0.000 | ******.........................*..****X | 10.502 | | 0.000 | *****...*.........*.****..*.*.***.******X | 7.100 | | 6.800 | ................***X | 6.800 | | 4.800 | ...........X | 4.800 | | 1.600 | ...X | 1.600 | | 0.000 | ****......*.....****X | 4.500 | | 0.300 | X | 0.300 | | 0.000 | ****.*.....*****..******X | 3.402 | | 0.000 | *****....................*.......****X | 10.500 | | 3.500 | ........X | 3.500 | | 3.000 | ......*****......................................*****........................................*...............****X | 37.800 | | 20.096 | ....................................................*****X | 20.096 | | 1.500 | ..*****...............*.....****X | 9.000 | | 7.201 | ..................X | 7.201 | | 7.101 | .................*****......................................****X | 21.397 | | 17.000 | ...........................................X | 17.000 | | 10.999 | ............................X | 10.999 | | 5.999 | ..............****.*...****X | 7.502 | | 2.000 | ....X | 2.000 | | 0.000 | ****************...*........****X | 4.501 | | 4.500 | ..........*...****X | 5.399 | | 5.000 | ............X | 5.000 | | 3.600 | ........****X | 3.600 | | 1.800 | ...X | 1.800 | | 0.100 | *****................*......................****...X | 15.804 | | 13.800 | ...................................******...........****X | 17.897 | | 4.419 | ..........X | 4.419 | | 3.219 | .......****X | 3.219 | | 1.500 | ..******............******............*****X | 10.500 | | 0.000 | ************************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 05:52:51,223 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 73.0871634439 | 6437847040 | 4323680256 | 2426556416 / 7511588864 | 101897 / 203788 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 81.3253112448 | 6437847040 | 4233875456 | 2466000896 / 7511588864 | 100109 / 200756 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 78.6268179852 | 6437847040 | 4238503936 | 2470383616 / 7511588864 | 100292 / 201430 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 64.2963086409 | 6437847040 | 3846664192 | 3062792192 / 9530753024 | 101704 / 199789 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 80.5739808673 | 6437847040 | 4286701568 | 2433982464 / 7511588864 | 101493 / 205165 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 05:53:04,168 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 05:54:29,996 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162733192 | 267103854 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 161152544 | 282698482 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 442244008 | 433319276 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 05:54:29,996 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #19 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 05:54:40,101 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 05:54:40,101 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 05:54:40,101 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 05:55:14,298 | test | INFO | pool-21-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 33.996999979 seconds to complete 2021-05-17 05:55:14,305 | test | INFO | pool-21-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 05:57:19,365 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 05:57:19,365 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 05:57:19,365 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 05:57:27,407 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 06:10:33,612 | test | INFO | pool-21-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...*....*****...........................*.................*****...............................*..................****X | 242.812 | | 225.942 | ............................................................................................X | 225.942 | | 209.036 | .....................................................................................****X | 209.036 | | 191.151 | .............................................................................X | 191.151 | | 180.147 | .........................................................................*.****.*..*****.......X | 206.897 | | 187.299 | ............................................................................*****.........*****.......*......****X | 242.302 | | 241.514 | ..................................................................................................X | 241.514 | | 240.273 | .................................................................................................****X | 240.273 | | 239.854 | .................................................................................................X | 239.854 | | 233.349 | ...............................................................................................****X | 233.349 | | 215.484 | .......................................................................................*....****X | 223.773 | | 156.075 | ...............................................................*****......*..............*...........****...X | 238.567 | | 236.374 | ................................................................................................X | 236.374 | | 233.685 | ...............................................................................................***X | 233.685 | | 180.777 | .........................................................................X | 180.777 | | 56.586 | ......................****X | 56.586 | | 42.491 | ................X | 42.491 | | 36.993 | ..............****X | 36.993 | | 10.501 | ...X | 10.501 | | 2.000 | ****X | 2.000 | | 1.200 | *****X | 1.300 | | 1.200 | X | 1.200 | | 1.200 | *****X | 1.500 | | 1.400 | ****X | 1.400 | | 0.100 | X | 0.100 | | 0.000 | **********X | 1.725 | | 1.327 | X | 1.327 | | 1.200 | ****X | 1.200 | | 0.200 | *X | 0.200 | | 0.200 | ***************X | 1.700 | | 1.200 | X | 1.200 | | 0.900 | ***************X | 1.403 | | 1.101 | X | 1.101 | | 0.802 | ******.****X | 3.305 | | 2.804 | X | 2.804 | | 2.404 | *****........*........*****............*...........****X | 95.588 | | 80.964 | ................................X | 80.964 | | 72.364 | ............................X | 72.364 | | 65.596 | ..........................X | 65.596 | | 65.596 | ..........................**X | 65.596 | | 64.496 | .........................X | 64.496 | | 59.504 | .......................******..******..*****X | 69.399 | | 65.207 | .........................****X | 65.207 | | 57.707 | ......................X | 57.707 | | 50.094 | ...................****X | 50.094 | | 40.494 | ...............X | 40.494 | | 0.901 | ******.****X | 3.402 | | 1.300 | X | 1.300 | | 1.001 | *****.******X | 2.702 | | 2.300 | *****X | 2.300 | | 2.100 | ****X | 2.100 | | 1.700 | ******.*****X | 3.201 | | 2.801 | *****X | 2.801 | | 0.400 | ***********X | 2.200 | | 2.001 | ****X | 2.001 | | 1.100 | X | 1.100 | | 0.900 | *****.******X | 4.301 | | 3.902 | ****X | 3.902 | | 2.400 | X | 2.400 | | 2.100 | *****X | 2.100 | | 0.400 | *****.*****X | 4.100 | | 2.699 | X | 2.699 | | 2.100 | ****X | 2.100 | | 1.699 | *****X | 1.699 | | 1.300 | ****X | 1.300 | | 0.700 | ******.**.***X | 4.898 | | 2.998 | *X | 2.998 | | 1.600 | ***X | 1.600 | | 1.000 | X | 1.000 | | 0.600 | *****************.*****X | 4.300 | | 2.699 | X | 2.699 | | 1.899 | ****X | 1.899 | | 1.399 | *X | 1.399 | | 0.900 | ***********.*****X | 3.900 | | 3.099 | X | 3.099 | | 2.999 | ***X | 2.999 | | 2.499 | X | 2.499 | | 2.000 | *****.*****X | 3.899 | | 2.999 | X | 2.999 | | 1.800 | ******.***X | 2.999 | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:10:34,213 | test | INFO | pool-21-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****..................................******...............................................*****X | 545.126 | | 517.637 | ............................................................................******...............*****X | 614.108 | | 441.049 | .................................................................***X | 441.049 | | 441.049 | .................................................................*X | 441.049 | | 394.166 | ..........................................................****X | 394.166 | | 365.442 | .....................................................X | 365.442 | | 306.248 | ............................................*****..*........*****...*****.............*...........................*****X | 657.723 | | 345.821 | ..................................................****X | 345.821 | | 318.959 | ..............................................X | 318.959 | | 228.773 | .................................*****.......*....................*..........*X | 475.343 | | 475.343 | ......................................................................*X | 475.343 | | 456.528 | ...................................................................X | 456.528 | | 308.514 | .............................................****X | 308.514 | | 305.448 | ............................................*.****X | 312.275 | | 299.261 | ...........................................X | 299.261 | | 283.838 | .........................................****X | 283.838 | | 255.940 | .....................................X | 255.940 | | 215.154 | ...............................****.................*.................................................****X | 656.763 | | 606.463 | ..........................................................................................X | 606.463 | | 569.524 | ....................................................................................X | 569.524 | | 525.020 | .............................................................................****......X | 562.635 | | 557.138 | ..................................................................................*..***X | 569.926 | | 560.411 | ...................................................................................*...*****X | 581.682 | | 529.525 | ..............................................................................****X | 529.525 | | 497.343 | .........................................................................X | 497.343 | | 436.226 | ................................................................******.........****X | 495.389 | | 491.989 | ........................................................................X | 491.989 | | 422.232 | ..............................................................*****X | 422.232 | | 418.202 | .............................................................X | 418.202 | | 388.163 | .........................................................*****...................................*.......****X | 666.414 | | 475.612 | ......................................................................X | 475.612 | | 452.975 | ..................................................................***X | 452.975 | | 413.782 | .............................................................X | 413.782 | | 341.638 | ..................................................*****...............*..........***X | 510.785 | | 504.858 | ..........................................................................X | 504.858 | | 417.752 | .............................................................****X | 417.752 | | 415.287 | .............................................................*.....*****....*...........*****...*..****X | 584.329 | | 530.954 | ..............................................................................X | 530.954 | | 500.188 | ..........................................................................***X | 500.188 | | 483.971 | .......................................................................X | 483.971 | | 411.766 | ............................................................*****X | 411.766 | | 374.923 | .......................................................*****X | 374.923 | | 276.254 | ........................................***X | 276.254 | | 276.254 | ........................................**................****X | 380.799 | | 319.184 | ..............................................X | 319.184 | | 297.362 | ...........................................****X | 297.362 | | 286.962 | ..........................................X | 286.962 | | 260.514 | ......................................*****.....*..................*****..........................*****X | 590.456 | | 566.988 | ....................................................................................**X | 566.988 | | 566.988 | ....................................................................................***........****X | 621.345 | | 621.030 | ............................................................................................X | 621.030 | | 567.619 | ....................................................................................****X | 567.619 | | 552.553 | .................................................................................*..*X | 562.264 | | 562.264 | ...................................................................................***.....*...X | 619.056 | | 619.056 | ...........................................................................................*X | 619.056 | | 619.056 | ...........................................................................................*X | 619.056 | | 576.865 | .....................................................................................X | 576.865 | | 523.706 | .............................................................................****X | 523.706 | | 510.312 | ...........................................................................*......***X | 550.555 | | 550.555 | .................................................................................X | 550.555 | | 528.314 | ..............................................................................X | 528.314 | | 501.781 | ..........................................................................X | 501.781 | | 462.072 | ....................................................................***X | 462.072 | | 439.712 | ................................................................X | 439.712 | | 422.452 | ..............................................................*****.....**...****......*...*****...*.****X | 560.196 | | 541.346 | ................................................................................*****X | 541.346 | | 488.935 | ........................................................................******...................*****X | 614.523 | | 448.415 | ..................................................................*****X | 452.912 | | 443.914 | .................................................................*****.......*...*****.....****X | 544.900 | | 514.101 | ............................................................................X | 514.101 | | 477.515 | ......................................................................X | 477.515 | | 441.056 | .................................................................****......*......*......***X | 563.899 | | 497.805 | .........................................................................*X | 497.805 | | 468.540 | .....................................................................***X | 468.540 | | 458.306 | ...................................................................*.*****............*****X | 541.611 | | 464.439 | ....................................................................******..........*****X | 530.072 | | 474.088 | ......................................................................******........*****X | 532.269 | | 469.388 | .....................................................................X | 469.388 | | 465.779 | ....................................................................*****......*..*****..............*..****X | 621.421 | | 520.108 | .............................................................................X | 520.108 | | 510.842 | ...........................................................................*****..X | 524.144 | | 518.835 | ............................................................................*****....*.X | 551.896 | | 548.211 | .................................................................................****.*.*...**X | 585.902 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:10:34,884 | test | INFO | pool-21-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****..******.............******........................******...******......****X | 74.877 | | 72.475 | .............................................X | 72.475 | | 58.282 | ....................................*****..........X | 73.365 | | 67.866 | ..........................................*****....................****X | 98.769 | | 98.569 | ..............................................................X | 98.569 | | 91.856 | ..........................................................X | 91.856 | | 81.268 | ...................................................****....*.....*****..X | 98.781 | | 90.479 | .........................................................****X | 90.479 | | 85.292 | ......................................................*....*****......X | 101.777 | | 100.084 | ...............................................................*****..******.*.*****...*..............*****..X | 136.272 | | 125.374 | ...............................................................................*****....................****X | 154.892 | | 146.492 | .............................................................................................X | 146.492 | | 146.279 | .............................................................................................*****..X | 148.983 | | 137.483 | .......................................................................................****X | 137.483 | | 133.779 | .....................................................................................X | 133.779 | | 129.997 | ..................................................................................*****.X | 131.290 | | 130.885 | ...................................................................................X | 130.885 | | 125.384 | ...............................................................................***X | 125.384 | | 120.288 | ............................................................................*..*****X | 122.498 | | 114.619 | ........................................................................****X | 114.619 | | 114.219 | ........................................................................X | 114.219 | | 108.183 | ....................................................................*****X | 108.183 | | 107.753 | ....................................................................******.........*****X | 121.584 | | 106.770 | ...................................................................****X | 106.770 | | 98.515 | ..............................................................X | 98.515 | | 82.603 | ....................................................*****.*.****X | 85.721 | | 83.297 | ....................................................*..*****X | 86.606 | | 37.103 | ......................*****X | 37.103 | | 3.505 | .****X | 3.505 | | 2.400 | X | 2.400 | | 1.400 | ****X | 1.400 | | 1.299 | *..*****............................................*****X | 72.007 | | 62.198 | .......................................*X | 62.198 | | 62.198 | .......................................**X | 62.198 | | 55.572 | ..................................X | 55.572 | | 50.383 | ...............................*****......*........****X | 71.417 | | 69.999 | ............................................X | 69.999 | | 64.200 | ........................................****X | 64.200 | | 63.797 | ........................................*.......******.......*****X | 86.613 | | 21.909 | .............*****X | 21.909 | | 1.701 | ******.****X | 3.802 | | 3.000 | X | 3.000 | | 2.701 | *****X | 2.701 | | 1.400 | ***********..*****X | 4.201 | | 3.000 | *****X | 3.000 | | 2.300 | X | 2.300 | | 2.200 | ****X | 2.200 | | 0.200 | **********X | 0.500 | | 0.400 | ******..**X | 4.100 | | 4.100 | .**X | 4.100 | | 3.000 | X | 3.000 | | 2.600 | *****X | 2.600 | | 1.900 | ****X | 1.900 | | 0.900 | X | 0.900 | | 0.800 | *****.**.*********X | 4.399 | | 1.698 | X | 1.698 | | 0.699 | ****X | 0.699 | | 0.500 | ************.*.****X | 3.998 | | 2.299 | X | 2.299 | | 1.400 | ****X | 1.400 | | 0.599 | X | 0.599 | | 0.200 | ***********X | 1.400 | | 1.200 | ****.*****X | 2.399 | | 1.399 | X | 1.399 | | 0.300 | *****X | 0.300 | | 0.000 | ******.******.*****X | 3.999 | | 2.699 | X | 2.699 | | 1.400 | *****.*****X | 1.800 | | 1.700 | ***********X | 2.300 | | 1.899 | X | 1.899 | | 1.499 | ***X | 1.499 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:10:35,137 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 06:10:35,184 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 06:10:45,700 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 06:10:45,865 | test | INFO | pool-21-thread-26 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.288333333333 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 59.3256779054 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 31.7972734242 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 33.9061302043 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 37.9183333333 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 06:10:45,917 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 06:10:51,035 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 06:10:53,509 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 06:10:56,740 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.11 2021-05-17 06:11:01,823 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 4.98 2021-05-17 06:11:06,900 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.42 2021-05-17 06:11:11,976 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 9.86 2021-05-17 06:11:17,053 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 13.22 2021-05-17 06:11:22,200 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 17.66 2021-05-17 06:11:27,301 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 21.84 2021-05-17 06:11:32,365 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 25.35 2021-05-17 06:11:37,434 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 29.38 2021-05-17 06:11:42,548 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 32.79 2021-05-17 06:11:47,625 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 34.64 2021-05-17 06:11:52,733 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 38.87 2021-05-17 06:11:57,848 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 43.2 2021-05-17 06:12:02,924 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 46.41 2021-05-17 06:12:08,020 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 49.38 2021-05-17 06:12:10,492 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:10,493 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:13,105 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 52.97 2021-05-17 06:12:18,187 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 56.16 2021-05-17 06:12:21,969 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1679 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:21,970 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1679 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1613 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1783 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:23,278 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 59.9 2021-05-17 06:12:28,368 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 64.34 2021-05-17 06:12:33,946 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 06:12:34,299 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2042 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:34,299 | test | INFO | pool-21-thread-3 | [table_view:display:72] Keys failed in bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2042 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:39,040 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 69.59 2021-05-17 06:12:41,970 | test | INFO | pool-21-thread-27 | [table_view:display:72] Keys failed in default:_default:vcormBtng7zhLnNNlWw-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1386 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:42,140 | test | INFO | pool-21-thread-15 | [table_view:display:72] Keys failed in default:_default:oj6GalhFQ66r9C8692Z-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-649 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-563 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-570 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:44,160 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 72.46 2021-05-17 06:12:44,223 | test | INFO | pool-21-thread-16 | [table_view:display:72] Keys failed in default:_default:PTs-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1386 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:46,561 | test | INFO | pool-21-thread-20 | [table_view:display:72] Keys failed in default:_default:PTs-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-922 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:46,809 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in default:_default:uka6Hbp79FNkHakj-32-752000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1216 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1227 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1185 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:49,236 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 75.06 2021-05-17 06:12:51,188 | test | INFO | pool-21-thread-2 | [table_view:display:72] Keys failed in default:_default:h%_-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1193 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1216 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1227 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1272 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1185 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:51,625 | test | INFO | pool-21-thread-10 | [table_view:display:72] Keys failed in default:_default:SoSDXFqXIg4HMF0viXq-32-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1386 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:52,088 | test | INFO | pool-21-thread-23 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-603 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-557 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-646 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-522 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-532 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-683 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:54,328 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 77.55 2021-05-17 06:12:55,661 | test | INFO | pool-21-thread-8 | [table_view:display:72] Keys failed in default:_default:MaM1gk90FDP6XUjAsT-32-752000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1430 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:57,375 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in default:_default:AoexOkWRHYr1yHOZv-32-752000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-908 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-902 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-977 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-920 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-965 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-932 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-910 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:58,494 | test | INFO | pool-21-thread-15 | [table_view:display:72] Keys failed in default:_default:oj6GalhFQ66r9C8692Z-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-838 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-768 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-898 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-855 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-756 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-771 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-880 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:12:59,430 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-17 06:13:00,585 | test | INFO | pool-21-thread-16 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1006 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1009 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1010 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1176 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1121 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1134 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:01,296 | test | INFO | pool-21-thread-9 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-669 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-659 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-557 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-624 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-643 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-512 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-545 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-641 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-575 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-642 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-661 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-530 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:04,513 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 82.38 2021-05-17 06:13:04,999 | test | INFO | pool-21-thread-19 | [table_view:display:72] Keys failed in default:53Ml8jzeujA4-52-114000:GWyVkbc-52-272000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:05,000 | test | INFO | pool-21-thread-19 | [table_view:display:72] Keys failed in default:53Ml8jzeujA4-52-114000:GWyVkbc-52-272000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:06,045 | test | INFO | pool-21-thread-20 | [table_view:display:72] Keys failed in default:Z-25-848000:2NON85nc-25-949000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1144 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:07,328 | test | INFO | pool-21-thread-18 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1005 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1040 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1165 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1178 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1003 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:07,447 | test | INFO | pool-21-thread-11 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1159 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1018 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1030 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1132 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1144 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1034 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:09,448 | test | INFO | pool-21-thread-23 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-768 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-812 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-835 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-769 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-810 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-701 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-753 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-773 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-762 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-862 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-880 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:09,607 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 84.54 2021-05-17 06:13:12,423 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in default:_default:j923StI-32-751000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-501 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-642 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-693 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:13,332 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1159 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-983 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1018 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1128 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1151 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1041 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1030 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1043 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1034 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1133 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:13,895 | test | INFO | pool-21-thread-2 | [table_view:display:72] Keys failed in default:_default:h%_-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1390 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1433 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1392 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:14,553 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in default:_default:uka6Hbp79FNkHakj-32-752000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:14,732 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 87.36 2021-05-17 06:13:16,321 | test | INFO | pool-21-thread-16 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1247 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1347 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1259 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1239 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1260 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1271 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1340 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1200 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1234 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:18,105 | test | INFO | pool-21-thread-10 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-669 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-621 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-500 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-655 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-575 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-642 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-530 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-691 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-590 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:18,594 | test | INFO | pool-21-thread-15 | [table_view:display:72] Keys failed in default:_default:oj6GalhFQ66r9C8692Z-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-971 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-950 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:19,601 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in default:_default:uka6Hbp79FNkHakj-32-752000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-527 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-544 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-633 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-685 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-540 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-673 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:19,601 | test | INFO | pool-21-thread-27 | [table_view:display:72] Keys failed in default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-527 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-544 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-633 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-685 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-540 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-673 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:19,871 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 89.7 2021-05-17 06:13:24,436 | test | INFO | pool-21-thread-11 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1259 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1194 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1330 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:24,437 | test | INFO | pool-21-thread-18 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1259 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1194 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1330 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:24,979 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 92.03 2021-05-17 06:13:26,279 | test | INFO | pool-21-thread-9 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-748 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-813 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-873 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:26,371 | test | INFO | pool-21-thread-23 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-928 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-971 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-950 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:29,292 | test | INFO | pool-21-thread-20 | [table_view:display:72] Keys failed in default:Z-25-848000:2NON85nc-25-949000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1205 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1326 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1305 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1295 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1395 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1362 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1330 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1375 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1222 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:30,059 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 94.96 2021-05-17 06:13:31,778 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in default:_default:j923StI-32-751000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-706 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-715 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-705 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-889 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-736 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-868 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-831 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-751 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-772 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:33,091 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1182 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1205 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1326 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1218 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1362 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1330 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1198 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1265 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1375 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1222 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:35,138 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 97.83 2021-05-17 06:13:36,522 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in default:_default:itJ%FRHhoG3-32-752000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1117 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1108 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1062 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1098 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1087 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:39,190 | test | INFO | pool-21-thread-16 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1404 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1407 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1384 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1410 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1421 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1388 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:40,233 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: running, progress: 99.95 2021-05-17 06:13:40,579 | test | INFO | pool-21-thread-10 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-705 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-889 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-868 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-744 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-831 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-787 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:40,582 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in default:_default:uka6Hbp79FNkHakj-32-752000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-705 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-889 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-868 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-744 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-787 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:40,986 | test | INFO | pool-21-thread-27 | [table_view:display:72] Keys failed in default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-705 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-889 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-868 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-744 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-787 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:42,259 | test | INFO | pool-21-thread-19 | [table_view:display:72] Keys failed in default:53Ml8jzeujA4-52-114000:GWyVkbc-52-272000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:42,259 | test | INFO | pool-21-thread-19 | [table_view:display:72] Keys failed in default:53Ml8jzeujA4-52-114000:GWyVkbc-52-272000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-556 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:43,177 | test | INFO | pool-21-thread-18 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1405 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:43,891 | test | INFO | pool-21-thread-11 | [table_view:display:72] Keys failed in default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1405 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:45,260 | test | INFO | pool-21-thread-17 | [table_view:display:72] Keys failed in default:_default:SoSDXFqXIg4HMF0viXq-32-751000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-539 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-626 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-601 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-513 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-568 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-555 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-523 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-630 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-663 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-650 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-583 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-660 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:46,892 | test | INFO | pool-21-thread-20 | [table_view:display:72] Keys failed in default:Z-25-848000:2NON85nc-25-949000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1559 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1406 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1450 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1440 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1584 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1596 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:47,730 | test | INFO | pool-21-thread-9 | [table_view:display:72] Keys failed in default:_default:Ei0_SS-5OSA87-32-751000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-903 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-967 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-978 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-976 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:49,250 | test | INFO | pool-21-thread-2 | [table_view:display:72] Keys failed in default:_default:h%_-32-753000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-626 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-555 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-578 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-523 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:49,687 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1406 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1385 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 06:13:50,369 | test | INFO | pool-21-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 06:13:50,476 | test | INFO | pool-21-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 184.611000061 sec 2021-05-17 06:14:22,398 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 06:14:22,400 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 06:19:10,529 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 06:19:10,530 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 06:19:10,530 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 06:19:10,532 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 4 docs from 500 to 980 2021-05-17 06:19:10,532 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-17 06:19:10,532 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 16 docs from 980 to 1440 2021-05-17 06:19:10,533 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 20 docs from 500 to 980 2021-05-17 06:19:10,535 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 23 docs from 980 to 1440 2021-05-17 06:19:10,536 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 22 docs from 980 to 1440 2021-05-17 06:19:10,538 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 29 docs from 500 to 980 2021-05-17 06:19:10,538 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 25 docs from 500 to 980 2021-05-17 06:20:02,096 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 06:20:02,098 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 35 docs from 980 to 1440 2021-05-17 06:20:02,099 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 29 docs from 500 to 980 2021-05-17 06:20:02,101 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 39 docs from 1000 to 1960 2021-05-17 06:20:02,101 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 29 docs from 980 to 1440 2021-05-17 06:20:02,101 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 28 docs from 500 to 980 2021-05-17 06:20:02,101 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 30 docs from 980 to 1440 2021-05-17 06:20:02,102 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 21 docs from 500 to 980 2021-05-17 06:20:02,102 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 20 docs from 500 to 980 2021-05-17 06:20:02,102 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 7 docs from 980 to 1440 2021-05-17 06:20:02,104 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 15 docs from 500 to 980 2021-05-17 06:20:17,809 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 06:29:13,552 | test | INFO | pool-21-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****...............................*................................***X | 98.200 | | 49.475 | ...............................X | 49.475 | | 2.299 | ***X | 2.399 | | 0.300 | ****...................*......****.X | 40.285 | | 39.896 | ........................****...............................*..............................**X | 132.897 | | 72.503 | ..............................................X | 72.503 | | 17.903 | ..........****....*.***.*.......****X | 38.303 | | 34.297 | .....................****..........*..****.......*****X | 64.002 | | 62.298 | .......................................****..*.***X | 67.212 | | 57.896 | ....................................***X | 57.896 | | 55.997 | ...................................X | 55.997 | | 51.604 | ................................*****........................................*..............****X | 134.800 | | 77.589 | .................................................X | 77.589 | | 56.996 | ....................................****X | 56.996 | | 37.701 | .......................X | 37.701 | | 23.699 | ..............***X | 23.699 | | 17.098 | ..........X | 17.098 | | 2.999 | *****X | 2.999 | | 1.800 | ******...........................*........****X | 56.701 | | 45.297 | ............................X | 45.297 | | 40.399 | .........................*****............................*........***X | 96.901 | | 72.101 | .............................................X | 72.101 | | 1.400 | *****....*.......*...****.......*..........................******.........*****X | 87.214 | | 68.698 | ...........................................**********X | 69.078 | | 50.496 | ...............................X | 50.496 | | 47.111 | .............................*****............*..****..*..*....****.X | 82.908 | | 72.507 | ..............................................******.....*......***X | 90.106 | | 88.500 | ........................................................X | 88.500 | | 85.203 | ......................................................******...........*****.*....................*....***X | 141.484 | | 137.110 | ........................................................................................X | 137.110 | | 127.123 | .................................................................................****..X | 129.827 | | 126.197 | ................................................................................*****.....*....***X | 139.921 | | 130.424 | ...................................................................................X | 130.424 | | 121.272 | .............................................................................X | 121.272 | | 106.271 | ....................................................................****....*.....*****....*................*****..X | 153.899 | | 136.698 | .......................................................................................*****....*.....****..X | 153.648 | | 135.168 | ......................................................................................****X | 135.168 | | 123.421 | ...............................................................................X | 123.421 | | 107.143 | ....................................................................****X | 107.143 | | 103.725 | ..................................................................X | 103.725 | | 81.316 | ...................................................X | 81.316 | | 80.402 | ...................................................****.........*............*....***X | 119.594 | | 118.922 | ............................................................................*...****X | 123.196 | | 123.002 | ..............................................................................X | 123.002 | | 116.873 | ..........................................................................****...*.............****X | 140.754 | | 130.759 | ...................................................................................X | 130.759 | | 81.503 | ...................................................****X | 81.503 | | 56.098 | ...................................X | 56.098 | | 1.200 | ***********X | 1.400 | | 0.800 | X | 0.800 | | 0.400 | ****.*......*...****.........*................******.............................................****X | 123.503 | | 90.603 | .........................................................X | 90.603 | | 0.600 | *****X | 0.600 | | 0.600 | *****........*...*****...................*......****X | 55.495 | | 53.902 | ..................................*.****.....*............*****X | 81.898 | | 79.494 | ..................................................*.****X | 80.604 | | 76.608 | ................................................*.****X | 78.104 | | 68.497 | ...........................................X | 68.497 | | 61.195 | ......................................****X | 61.195 | | 50.097 | ...............................*.*****...X | 56.398 | | 54.693 | ..................................*****X | 54.812 | | 46.012 | ............................****X | 46.012 | | 32.202 | ...................X | 32.202 | | 22.081 | .............****X | 22.081 | | 6.001 | ..X | 6.001 | | 2.200 | ****X | 2.200 | | 1.100 | X | 1.100 | | 0.600 | ****X | 0.600 | | 0.300 | X | 0.300 | | 0.000 | *****************..........................******.****X | 42.059 | | 31.556 | ...................X | 31.556 | | 0.600 | *****X | 0.600 | | 0.400 | ******.****X | 1.699 | | 1.398 | X | 1.398 | | 0.499 | ****X | 0.499 | | 0.200 | *****...*................******......................*****X | 63.442 | | 5.000 | ..****X | 5.000 | | 1.300 | X | 1.300 | | 0.400 | **********.*****X | 1.699 | | 0.600 | ***********..........*..................................****X | 68.165 | | 45.965 | ............................X | 45.965 | | 12.592 | .......***X | 12.592 | | 7.293 | ...X | 7.293 | | 4.195 | .X | 4.195 | | 1.000 | ****X | 1.000 | | 0.800 | X | 0.800 | | 0.400 | ****X | 0.400 | | 0.000 | ******...................*****...........*....****X | 53.174 | | 34.572 | .....................X | 34.572 | | 18.889 | ...........***X | 18.889 | | 14.891 | ........X | 14.891 | | 1.400 | *****X | 1.400 | | 0.000 | ***********..*..........*****.......*...................****X | 58.676 | | 44.277 | ...........................X | 44.277 | | 4.600 | .****X | 4.600 | | 3.399 | .X | 3.399 | | 2.899 | X | 2.899 | | 0.400 | ****X | 0.400 | | 0.300 | X | 0.300 | | 0.100 | **X | 0.100 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:29:14,558 | test | INFO | pool-21-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................****..................................................................................X | 1089.074 | | 971.656 | ........................................................................................***X | 971.656 | | 494.372 | ............................................X | 494.372 | | 316.503 | ............................****.***X | 326.869 | | 280.987 | ........................X | 280.987 | | 197.463 | .................***X | 197.463 | | 197.052 | .................X | 197.052 | | 187.969 | ................****X | 194.624 | | 171.005 | ..............**X | 171.005 | | 142.747 | ............*.****..............*........***X | 401.490 | | 273.330 | ........................X | 273.330 | | 255.035 | ......................**X | 255.035 | | 241.760 | .....................X | 241.760 | | 237.772 | ....................***X | 237.772 | | 237.058 | ....................*.****.*.***X | 272.009 | | 172.702 | ..............X | 172.702 | | 139.199 | ...........****..*...****..*.***X | 224.788 | | 129.788 | ..........X | 129.788 | | 103.707 | ........*****.*****....*...........****X | 276.478 | | 267.206 | .......................*..*.****..............*..............****X | 603.970 | | 457.615 | .........................................X | 457.615 | | 374.752 | .................................X | 374.752 | | 291.899 | .........................***X | 291.899 | | 252.716 | ......................X | 252.716 | | 229.305 | ....................*****X | 231.488 | | 225.190 | ...................****X | 225.190 | | 194.592 | ................******...................................X | 568.715 | | 568.715 | ...................................................***X | 568.715 | | 421.458 | .....................................X | 421.458 | | 378.358 | .................................X | 378.358 | | 331.271 | .............................*X | 331.271 | | 331.271 | .............................*X | 331.271 | | 266.591 | .......................X | 266.591 | | 170.783 | ..............******.****X | 178.532 | | 151.500 | ............X | 151.500 | | 142.796 | ............*****X | 152.170 | | 148.753 | ............*****.....*..****..*...........****X | 362.660 | | 356.645 | ...............................X | 356.645 | | 264.377 | .......................*****X | 264.377 | | 188.914 | ................*****X | 188.914 | | 184.512 | ...............******.****X | 191.805 | | 184.687 | ...............*.**X | 187.691 | | 187.691 | ................X | 187.691 | | 183.207 | ...............*.....*.****....*..***X | 314.057 | | 307.964 | ...........................X | 307.964 | | 276.193 | ........................*********X | 282.328 | | 247.032 | .....................X | 247.032 | | 176.293 | ...............X | 176.293 | | 141.393 | ...........****..*...*****..*......*****X | 276.300 | | 175.795 | ...............*****.*......*****X | 255.758 | | 223.451 | ...................*X | 223.451 | | 223.451 | ...................**X | 223.451 | | 211.921 | ..................X | 211.921 | | 210.092 | ..................****X | 210.092 | | 204.739 | .................X | 204.739 | | 199.805 | .................*****........*...****X | 317.389 | | 267.408 | .......................X | 267.408 | | 245.294 | .....................**X | 245.294 | | 245.294 | .....................*X | 245.294 | | 217.468 | ..................*****X | 217.468 | | 179.582 | ...............*****......*.........*****...*..****X | 393.509 | | 362.291 | ................................X | 362.291 | | 356.282 | ...............................*****....*.****X | 406.188 | | 321.801 | ............................X | 321.801 | | 247.018 | .....................****X | 247.018 | | 187.728 | ................****X | 187.728 | | 181.723 | ...............******.*...*****...*........****X | 342.643 | | 283.598 | .........................X | 283.598 | | 209.132 | ..................****X | 209.132 | | 196.379 | .................X | 196.379 | | 177.982 | ...............*****..*...******X | 238.524 | | 232.095 | ....................X | 232.095 | | 232.095 | ....................**X | 232.095 | | 203.684 | .................X | 203.684 | | 174.709 | ...............****X | 174.709 | | 166.518 | ..............X | 166.518 | | 159.408 | .............******.****X | 174.187 | | 157.383 | .............X | 157.383 | | 144.598 | ............****X | 144.598 | | 134.614 | ...........*****X | 139.881 | | 138.597 | ...........*..*****.....*..*..***X | 253.394 | | 244.418 | .....................*.*****.*****X | 267.289 | | 241.040 | .....................X | 241.040 | | 221.038 | ...................****X | 221.038 | | 200.369 | .................*****...*.............*****X | 374.126 | | 272.158 | .......................X | 272.158 | | 249.649 | .....................***X | 249.649 | | 237.766 | ....................X | 237.766 | | 231.663 | ....................****X | 231.663 | | 211.171 | ..................X | 211.171 | | 184.734 | ...............***X | 184.734 | | 183.539 | ...............*.X | 188.321 | | 180.628 | ...............****..*.....****X | 256.055 | | 247.067 | .....................X | 247.067 | | 221.386 | ...................*****...*.....*****..X | 330.024 | | 321.052 | ............................****X | 321.052 | | 197.056 | .................X | 197.056 | | 169.845 | ..............****.*******...******.......*..........***X | 396.001 | | 388.284 | ..................................X | 388.284 | | 354.355 | ...............................****.******X | 362.480 | | 354.176 | ...............................****X | 354.176 | | 313.080 | ...........................X | 313.080 | | 241.264 | .....................****X | 241.264 | | 240.043 | .....................X | 240.043 | | 203.265 | .................***X | 203.265 | | 194.957 | ................*.*****..*.........****X | 316.848 | | 255.080 | ......................X | 255.080 | | 228.986 | ....................*********X | 233.002 | | 174.736 | ...............******.*.....****.....*.......*****.*...*.***X | 433.191 | | 392.560 | ...................................X | 392.560 | | 340.065 | ..............................X | 340.065 | | 309.153 | ...........................***X | 309.153 | | 280.965 | ........................X | 280.965 | | 224.521 | ...................X | 224.521 | | 197.121 | .................****X | 206.416 | | 204.625 | .................******.............*****X | 347.286 | | 295.919 | ..........................*****.........*..............***X | 552.101 | +---------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 06:29:14,688 | test | INFO | pool-21-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................****.................................................................X | 193.503 | | 147.336 | ................................................................***X | 147.336 | | 50.585 | .....................X | 50.585 | | 10.494 | ...***..*...****X | 20.607 | | 12.401 | ....***X | 12.401 | | 12.299 | ....*..****....*.....***X | 38.098 | | 29.690 | ............X | 29.690 | | 20.504 | ........****....*...***.*...*****....****.....................*.........***X | 122.680 | | 94.809 | .........................................***X | 94.809 | | 69.300 | .............................*......****X | 81.498 | | 73.211 | ...............................****...*....***X | 88.194 | | 82.494 | ...................................X | 82.494 | | 66.705 | ............................*****........X | 83.808 | | 82.801 | ...................................*****....*...*****..*..*****..................................................****X | 219.926 | | 191.667 | ...................................................................................X | 191.667 | | 154.096 | ...................................................................*****X | 154.096 | | 126.505 | .......................................................*****X | 126.505 | | 112.392 | ................................................*****..X | 115.985 | | 109.599 | ...............................................*****.......................................*...........*****.*****X | 225.652 | | 151.198 | ..................................................................X | 151.198 | | 75.692 | ................................*****....*....***X | 93.417 | | 83.511 | ....................................X | 83.511 | | 72.998 | ...............................*****X | 73.298 | | 61.088 | ..........................X | 61.088 | | 51.685 | .....................***X | 51.685 | | 48.187 | ....................X | 48.187 | | 41.898 | .................*.***X | 44.808 | | 38.704 | ................X | 38.704 | | 26.807 | ..........X | 26.807 | | 18.303 | .......****..*........******......****X | 55.803 | | 48.404 | ....................X | 48.404 | | 39.303 | ................*****..*.****X | 47.001 | | 22.598 | .........X | 22.598 | | 15.499 | .....***X | 15.499 | | 11.300 | ....X | 11.300 | | 3.100 | X | 3.100 | | 0.000 | ***************X | 1.200 | | 0.800 | ******..........................****X | 59.737 | | 47.596 | ....................X | 47.596 | | 36.899 | ...............****X | 36.899 | | 17.799 | ......X | 17.799 | | 3.000 | *****X | 3.200 | | 1.400 | ****X | 1.400 | | 0.300 | X | 0.300 | | 0.000 | **********...*..........*****....*.....*****.....*................****X | 99.096 | | 47.112 | ...................X | 47.112 | | 0.400 | ****X | 0.400 | | 0.300 | *****X | 0.400 | | 0.200 | ************.*****X | 2.400 | | 2.000 | *****..*.......*****..........*...........****X | 68.305 | | 58.304 | ........................X | 58.304 | | 51.807 | .....................*****...******....*...*****....X | 82.208 | | 81.808 | ...................................****X | 81.808 | | 74.516 | ................................X | 74.516 | | 71.713 | ..............................****X | 71.713 | | 63.389 | ...........................X | 63.389 | | 59.294 | .........................****X | 59.294 | | 24.001 | .........*****X | 24.001 | | 0.400 | *****X | 0.400 | | 0.200 | *****....*.............*****......*......................****X | 103.792 | | 98.990 | ..........................................X | 98.990 | | 86.796 | .....................................X | 86.796 | | 82.001 | ...................................***X | 82.001 | | 77.909 | .................................X | 77.909 | | 65.599 | ............................X | 65.599 | | 57.393 | ........................***X | 57.393 | | 53.498 | ......................X | 53.498 | | 48.611 | ....................******..........******.......****X | 85.904 | | 78.997 | ..................................X | 78.997 | | 72.997 | ...............................*****.....*..****X | 88.475 | | 76.673 | ................................X | 76.673 | | 62.733 | ..........................X | 62.733 | | 58.231 | ........................***X | 58.231 | | 43.831 | ..................X | 43.831 | | 14.600 | .....X | 14.600 | | 0.500 | ******.*****......*****...................*......****.X | 75.739 | | 75.331 | ................................*...***X | 81.311 | | 69.509 | .............................X | 69.509 | | 37.067 | ...............*****.........*..........****X | 79.017 | | 31.792 | .............X | 31.792 | | 12.193 | ....***X | 12.193 | | 8.094 | ..X | 8.094 | | 0.599 | *****...*.............******....................****X | 81.741 | | 77.669 | .................................X | 77.669 | | 72.948 | ...............................****X | 72.948 | | 9.392 | ...****X | 9.392 | | 5.091 | .X | 5.091 | | 0.999 | **********X | 1.400 | | 0.800 | X | 0.800 | | 0.800 | **********.......*.......*................****...*.......****X | 90.455 | | 70.761 | ..............................X | 70.761 | | 4.898 | .*****............*.................****X | 70.168 | | 33.986 | ..............******.*.*****.*****X | 42.575 | | 2.400 | *****...*............*****...........*.........****X | 81.562 | | 72.674 | ...............................X | 72.674 | | 64.277 | ...........................****X | 64.277 | | 60.575 | .........................X | 60.575 | | 60.477 | .........................X | 60.477 | | 59.175 | .........................***X | 59.175 | | 49.976 | .....................X | 49.976 | | 23.893 | .........*****..................*......****X | 76.976 | | 24.101 | .........X | 24.101 | | 4.199 | ***X | 4.199 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:29:14,750 | test | WARNING | Thread-66148 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:2vs61CFqhUHQE6S0Oh-50-490000:COA33LI--50-680000 create failed keys from task: [u'test_collections-1270', u'test_collections-1679', u'test_collections-1613', u'test_collections-2042', u'test_collections-1783'] 2021-05-17 06:29:14,760 | test | WARNING | Thread-66190 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:SoSDXFqXIg4HMF0viXq-32-751000 update failed keys from task: [u'test_collections-1391', u'test_collections-1401', u'test_collections-1386'] 2021-05-17 06:29:14,762 | test | WARNING | Thread-66191 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:oj6GalhFQ66r9C8692Z-32-753000 delete failed keys from task: [u'test_collections-728', u'test_collections-649', u'test_collections-823', u'test_collections-768', u'test_collections-887', u'test_collections-767', u'test_collections-863', u'test_collections-563', u'test_collections-880', u'test_collections-838', u'test_collections-836', u'test_collections-898', u'test_collections-855', u'test_collections-756', u'test_collections-954', u'test_collections-971', u'test_collections-950', u'test_collections-892', u'test_collections-771', u'test_collections-570'] 2021-05-17 06:29:14,763 | test | WARNING | Thread-66196 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:j923StI-32-751000 delete failed keys from task: [u'test_collections-609', u'test_collections-889', u'test_collections-844', u'test_collections-885', u'test_collections-642', u'test_collections-562', u'test_collections-760', u'test_collections-736', u'test_collections-535', u'test_collections-613', u'test_collections-852', u'test_collections-897', u'test_collections-772', u'test_collections-693', u'test_collections-706', u'test_collections-828', u'test_collections-705', u'test_collections-868', u'test_collections-501', u'test_collections-740', u'test_collections-715', u'test_collections-831', u'test_collections-597', u'test_collections-751', u'test_collections-592'] 2021-05-17 06:29:14,765 | test | WARNING | Thread-66200 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Ei0_SS-5OSA87-32-751000 update failed keys from task: [u'test_collections-1347', u'test_collections-1109', u'test_collections-1260', u'test_collections-1261', u'test_collections-1340', u'test_collections-1384', u'test_collections-1341', u'test_collections-1220', u'test_collections-1421', u'test_collections-1388', u'test_collections-1016', u'test_collections-1336', u'test_collections-1259', u'test_collections-1019', u'test_collections-1010', u'test_collections-1176', u'test_collections-1099', u'test_collections-1134', u'test_collections-1410', u'test_collections-1247', u'test_collections-1006', u'test_collections-980', u'test_collections-1129', u'test_collections-1404', u'test_collections-1009', u'test_collections-1407', u'test_collections-1121', u'test_collections-1045', u'test_collections-1200', u'test_collections-1081', u'test_collections-1239', u'test_collections-1271', u'test_collections-1351', u'test_collections-1111', u'test_collections-1234'] 2021-05-17 06:29:14,766 | test | WARNING | Thread-66206 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:PTs-32-753000 update failed keys from task: [u'test_collections-1401', u'test_collections-1386'] 2021-05-17 06:29:14,767 | test | WARNING | Thread-66209 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:AoexOkWRHYr1yHOZv-32-752000 delete failed keys from task: [u'test_collections-908', u'test_collections-917', u'test_collections-902', u'test_collections-977', u'test_collections-920', u'test_collections-965', u'test_collections-932', u'test_collections-910', u'test_collections-952'] 2021-05-17 06:29:14,763 | test | WARNING | Thread-66197 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:MaM1gk90FDP6XUjAsT-32-752000 update failed keys from task: [u'test_collections-1430'] 2021-05-17 06:29:14,779 | test | WARNING | Thread-66213 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:uka6Hbp79FNkHakj-32-752000 update failed keys from task: [u'test_collections-1190', u'test_collections-1292', u'test_collections-1368', u'test_collections-1302', u'test_collections-1227', u'test_collections-1307', u'test_collections-1428', u'test_collections-1382', u'test_collections-1185', u'test_collections-1320', u'test_collections-1244', u'test_collections-1235', u'test_collections-1412', u'test_collections-1216', u'test_collections-1373', u'test_collections-1297'] 2021-05-17 06:29:14,767 | test | WARNING | Thread-66210 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 update failed keys from task: [u'test_collections-1262', u'test_collections-1385', u'test_collections-1265', u'test_collections-1222', u'test_collections-1267', u'test_collections-1182', u'test_collections-1138', u'test_collections-1018', u'test_collections-1218', u'test_collections-1055', u'test_collections-1330', u'test_collections-1133', u'test_collections-1375', u'test_collections-983', u'test_collections-1128', u'test_collections-1205', u'test_collections-1326', u'test_collections-1129', u'test_collections-1406', u'test_collections-1041', u'test_collections-1043', u'test_collections-1362', u'test_collections-1241', u'test_collections-1159', u'test_collections-1151', u'test_collections-1030', u'test_collections-1031', u'test_collections-1198', u'test_collections-1034', u'test_collections-1353'] 2021-05-17 06:29:14,779 | test | WARNING | Thread-66211 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:itJ%FRHhoG3-32-752000 update failed keys from task: [u'test_collections-1028', u'test_collections-1117', u'test_collections-1008', u'test_collections-1108', u'test_collections-1062', u'test_collections-1098', u'test_collections-1087'] 2021-05-17 06:29:14,819 | test | WARNING | Thread-66206 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:PTs-32-753000 delete failed keys from task: [u'test_collections-939', u'test_collections-948', u'test_collections-947', u'test_collections-922'] 2021-05-17 06:29:14,819 | test | WARNING | Thread-66190 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:SoSDXFqXIg4HMF0viXq-32-751000 delete failed keys from task: [u'test_collections-627', u'test_collections-626', u'test_collections-601', u'test_collections-568', u'test_collections-566', u'test_collections-523', u'test_collections-663', u'test_collections-583', u'test_collections-660', u'test_collections-539', u'test_collections-513', u'test_collections-632', u'test_collections-555', u'test_collections-630', u'test_collections-650'] 2021-05-17 06:29:15,068 | test | WARNING | Thread-66245 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 update failed keys from task: [u'test_collections-1106', u'test_collections-1109', u'test_collections-1261', u'test_collections-1220', u'test_collections-1144', u'test_collections-1267', u'test_collections-1336', u'test_collections-1259', u'test_collections-1018', u'test_collections-1096', u'test_collections-1099', u'test_collections-1132', u'test_collections-1055', u'test_collections-1330', u'test_collections-988', u'test_collections-982', u'test_collections-980', u'test_collections-1204', u'test_collections-1129', u'test_collections-1405', u'test_collections-1045', u'test_collections-1081', u'test_collections-1159', u'test_collections-1237', u'test_collections-1194', u'test_collections-1030', u'test_collections-1351', u'test_collections-1111', u'test_collections-1034'] 2021-05-17 06:29:15,101 | test | WARNING | Thread-66225 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h%_-32-753000 update failed keys from task: [u'test_collections-1190', u'test_collections-1390', u'test_collections-1193', u'test_collections-1302', u'test_collections-1423', u'test_collections-1227', u'test_collections-1307', u'test_collections-1183', u'test_collections-1185', u'test_collections-1216', u'test_collections-1373', u'test_collections-1297', u'test_collections-1292', u'test_collections-1368', u'test_collections-1320', u'test_collections-1244', u'test_collections-1235', u'test_collections-1433', u'test_collections-1435', u'test_collections-1392', u'test_collections-1272', u'test_collections-1230', u'test_collections-1275'] 2021-05-17 06:29:15,114 | test | WARNING | Thread-66227 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Z-25-848000:2NON85nc-25-949000 update failed keys from task: [u'test_collections-1305', u'test_collections-1109', u'test_collections-1261', u'test_collections-1262', u'test_collections-1186', u'test_collections-1144', u'test_collections-1584', u'test_collections-1189', u'test_collections-1222', u'test_collections-1267', u'test_collections-1016', u'test_collections-1414', u'test_collections-1019', u'test_collections-1295', u'test_collections-1450', u'test_collections-1099', u'test_collections-1055', u'test_collections-1330', u'test_collections-1375', u'test_collections-1521', u'test_collections-1566', u'test_collections-1522', u'test_collections-1204', u'test_collections-1567', u'test_collections-1205', u'test_collections-1326', u'test_collections-1129', u'test_collections-1406', u'test_collections-1362', u'test_collections-1241', u'test_collections-1440', u'test_collections-1045', u'test_collections-1081', u'test_collections-1559', u'test_collections-1394', u'test_collections-1395', u'test_collections-1111', u'test_collections-1353', u'test_collections-1596'] 2021-05-17 06:29:15,114 | test | WARNING | Thread-66229 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:53Ml8jzeujA4-52-114000:GWyVkbc-52-272000 create failed keys from task: [u'test_collections-556', u'test_collections-156'] 2021-05-17 06:29:15,121 | test | WARNING | Thread-66249 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 update failed keys from task: [u'test_collections-1149', u'test_collections-1261', u'test_collections-1220', u'test_collections-1267', u'test_collections-1015', u'test_collections-1059', u'test_collections-1336', u'test_collections-1259', u'test_collections-1172', u'test_collections-1330', u'test_collections-1178', u'test_collections-1005', u'test_collections-1204', u'test_collections-1405', u'test_collections-1040', u'test_collections-1164', u'test_collections-1165', u'test_collections-1000', u'test_collections-1003', u'test_collections-1237', u'test_collections-1194', u'test_collections-1351'] 2021-05-17 06:29:15,137 | test | WARNING | Thread-66218 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:vcormBtng7zhLnNNlWw-32-753000 update failed keys from task: [u'test_collections-1401', u'test_collections-1386'] 2021-05-17 06:29:15,711 | test | WARNING | Thread-66213 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:uka6Hbp79FNkHakj-32-752000 delete failed keys from task: [u'test_collections-708', u'test_collections-828', u'test_collections-705', u'test_collections-889', u'test_collections-527', u'test_collections-868', u'test_collections-744', u'test_collections-844', u'test_collections-544', u'test_collections-885', u'test_collections-787', u'test_collections-685', u'test_collections-740', u'test_collections-540', u'test_collections-760', u'test_collections-678', u'test_collections-852', u'test_collections-633', u'test_collections-597', u'test_collections-673', u'test_collections-690'] 2021-05-17 06:29:16,043 | test | WARNING | Thread-66210 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:qqf8QpMz-Gg6pdsxCzZ-32-753000 delete failed keys from task: [u'test_collections-828', u'test_collections-705', u'test_collections-889', u'test_collections-527', u'test_collections-868', u'test_collections-744', u'test_collections-844', u'test_collections-544', u'test_collections-885', u'test_collections-787', u'test_collections-685', u'test_collections-740', u'test_collections-540', u'test_collections-760', u'test_collections-678', u'test_collections-852', u'test_collections-633', u'test_collections-597', u'test_collections-673', u'test_collections-690'] 2021-05-17 06:29:16,663 | test | WARNING | Thread-66225 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h%_-32-753000 delete failed keys from task: [u'test_collections-626', u'test_collections-566', u'test_collections-555', u'test_collections-578', u'test_collections-523'] 2021-05-17 06:29:16,720 | test | WARNING | Thread-66245 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XsXsTSPjaIYAa-32-762000:diFnv8W_6Bgj9-32-764000 delete failed keys from task: [u'test_collections-609', u'test_collections-889', u'test_collections-844', u'test_collections-885', u'test_collections-642', u'test_collections-562', u'test_collections-760', u'test_collections-535', u'test_collections-613', u'test_collections-852', u'test_collections-655', u'test_collections-575', u'test_collections-530', u'test_collections-691', u'test_collections-708', u'test_collections-828', u'test_collections-705', u'test_collections-669', u'test_collections-868', u'test_collections-744', u'test_collections-621', u'test_collections-500', u'test_collections-787', u'test_collections-740', u'test_collections-831', u'test_collections-631', u'test_collections-592', u'test_collections-590'] 2021-05-17 06:29:16,753 | test | WARNING | Thread-66249 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:XsXsTSPjaIYAa-32-762000:wvNLCA9QMQe-32-764000 delete failed keys from task: [u'test_collections-928', u'test_collections-606', u'test_collections-603', u'test_collections-768', u'test_collections-769', u'test_collections-646', u'test_collections-767', u'test_collections-522', u'test_collections-762', u'test_collections-683', u'test_collections-880', u'test_collections-818', u'test_collections-812', u'test_collections-810', u'test_collections-773', u'test_collections-971', u'test_collections-532', u'test_collections-653', u'test_collections-507', u'test_collections-866', u'test_collections-701', u'test_collections-862', u'test_collections-541', u'test_collections-783', u'test_collections-835', u'test_collections-557', u'test_collections-799', u'test_collections-753', u'test_collections-950'] 2021-05-17 06:29:16,809 | test | WARNING | Thread-66200 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Ei0_SS-5OSA87-32-751000 delete failed keys from task: [u'test_collections-966', u'test_collections-967', u'test_collections-643', u'test_collections-641', u'test_collections-642', u'test_collections-562', u'test_collections-659', u'test_collections-813', u'test_collections-857', u'test_collections-978', u'test_collections-976', u'test_collections-575', u'test_collections-653', u'test_collections-530', u'test_collections-748', u'test_collections-903', u'test_collections-669', u'test_collections-624', u'test_collections-545', u'test_collections-584', u'test_collections-661', u'test_collections-783', u'test_collections-518', u'test_collections-557', u'test_collections-799', u'test_collections-512', u'test_collections-631', u'test_collections-873', u'test_collections-870'] 2021-05-17 06:29:28,217 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 06:30:31,039 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 06:30:31,039 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 06:30:55,997 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 06:30:55,999 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 06:31:17,177 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 06:31:17,177 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 06:32:19,572 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1470000744 seconds to complete 2021-05-17 06:32:19,586 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3789.567s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 19 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_19 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_20 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_20', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 20, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 06:32:29,009 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #20 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 06:32:29,690 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #20 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 06:32:30,115 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb471f5 type=client open_count=1 channel=[id: 0x65639f64, 0.0.0.0/0.0.0.0:37277] timeout=300.0>) 2021-05-17 06:32:50,828 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb471f6 type=client open_count=1 channel=[id: 0xdf74535d, 0.0.0.0/0.0.0.0:33239] timeout=300.0>) 2021-05-17 06:32:52,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb471f7 type=client open_count=1 channel=[id: 0x1e7180e4, 0.0.0.0/0.0.0.0:40590] timeout=300.0>) 2021-05-17 06:32:54,867 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb471f8 type=client open_count=1 channel=[id: 0x4e293543, 0.0.0.0/0.0.0.0:52061] timeout=300.0>) 2021-05-17 06:33:18,786 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb471f9 type=client open_count=1 channel=[id: 0xc7a05cc6, 0.0.0.0/0.0.0.0:60000] timeout=300.0>) 2021-05-17 06:33:20,802 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb471fa type=client open_count=1 channel=[id: 0x41493797, 0.0.0.0/0.0.0.0:43305] timeout=300.0>) 2021-05-17 06:33:22,821 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb471fb type=client open_count=1 channel=[id: 0x19791734, 0.0.0.0/0.0.0.0:55022] timeout=300.0>) 2021-05-17 06:33:47,015 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb471fc type=client open_count=1 channel=[id: 0x58dfc886, 0.0.0.0/0.0.0.0:43208] timeout=300.0>) 2021-05-17 06:33:49,032 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb471fd type=client open_count=1 channel=[id: 0x24e92b70, 0.0.0.0/0.0.0.0:51322] timeout=300.0>) 2021-05-17 06:33:51,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb471fe type=client open_count=1 channel=[id: 0x8fee3ca0, 0.0.0.0/0.0.0.0:35619] timeout=300.0>) 2021-05-17 06:34:15,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb471ff type=client open_count=1 channel=[id: 0x83cc2aad, 0.0.0.0/0.0.0.0:45994] timeout=300.0>) 2021-05-17 06:34:17,242 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb47200 type=client open_count=1 channel=[id: 0xf277c168, 0.0.0.0/0.0.0.0:57056] timeout=300.0>) 2021-05-17 06:34:19,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb47201 type=client open_count=1 channel=[id: 0x5264492f, 0.0.0.0/0.0.0.0:59200] timeout=300.0>) 2021-05-17 06:34:43,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb47202 type=client open_count=1 channel=[id: 0x0a403d8e, 0.0.0.0/0.0.0.0:45125] timeout=300.0>) 2021-05-17 06:34:45,681 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb47203 type=client open_count=1 channel=[id: 0x8a528eb8, 0.0.0.0/0.0.0.0:39137] timeout=300.0>) 2021-05-17 06:34:47,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 06:35:04,598 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 06:35:18,316 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 06:35:18,338 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #20 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 06:35:18,375 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #20 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 06:35:32,020 | test | INFO | pool-22-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.66166666667 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 06:35:32,043 | test | INFO | pool-22-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 06:35:42,115 | test | INFO | pool-22-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 06:35:42,141 | test | INFO | pool-22-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1210000515 sec 2021-05-17 06:35:42,403 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 06:36:25,559 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 06:36:25,868 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 779527496 | 52448602 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 691669392 | 83231156 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 1198893128 | 10882108 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 06:36:25,868 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 06:37:11,571 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 06:37:11,578 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 06:54:05,776 | test | INFO | pool-22-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***....................................................................................................****X | 79.831 | | 6.582 | .......X | 6.582 | | 1.203 | ****X | 1.203 | | 0.000 | ***********************************.*****..*..****X | 4.502 | | 1.700 | .*X | 1.700 | | 1.500 | ***X | 1.500 | | 0.100 | ******.X | 1.500 | | 1.500 | ****X | 1.500 | | 1.500 | X | 1.500 | | 1.401 | ****X | 1.401 | | 0.000 | ***********************.******..X | 2.901 | | 2.901 | ..****X | 2.901 | | 0.100 | X | 0.100 | | 0.000 | *****...*****X | 3.001 | | 0.000 | ***********....*.***X | 4.001 | | 0.600 | X | 0.600 | | 0.400 | *****.....*****X | 4.501 | | 1.500 | ***********..*****X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | **********************...*****X | 3.000 | | 1.600 | .X | 1.600 | | 1.500 | ****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | ****************.......*****X | 6.000 | | 0.000 | ***********************...******.****X | 3.400 | | 1.400 | *X | 1.400 | | 1.100 | ***X | 1.100 | | 0.800 | X | 0.800 | | 0.000 | ****************.....******..******......*.***X | 11.499 | | 6.098 | ......X | 6.098 | | 1.300 | X | 1.300 | | 0.500 | ****X | 0.500 | | 0.000 | ****************.......*****X | 5.799 | | 1.600 | .X | 1.600 | | 1.500 | ****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | **********..*...****X | 4.500 | | 0.300 | *X | 0.300 | | 0.000 | ****.......*.*****......****X | 11.505 | | 7.301 | ........X | 7.301 | | 5.900 | ......****X | 5.900 | | 3.800 | ...X | 3.800 | | 1.500 | ****.*.....*****X | 6.300 | | 6.000 | ......X | 6.000 | | 5.900 | ......****.......*****X | 11.800 | | 0.700 | X | 0.700 | | 0.000 | ****************.*****.......*.***X | 7.500 | | 5.500 | .....X | 5.500 | | 0.500 | X | 0.500 | | 0.000 | ****....*.*****....**.***X | 8.201 | | 5.200 | .....X | 5.200 | | 5.100 | .....******X | 5.500 | | 5.100 | .....****..*.****....****X | 10.599 | | 7.700 | ........X | 7.700 | | 0.700 | X | 0.700 | | 0.000 | ****.*****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | *****..*...***X | 4.500 | | 3.300 | ...X | 3.300 | | 0.300 | X | 0.300 | | 0.000 | *********************.......******...........*****X | 15.099 | | 5.301 | .....X | 5.301 | | 4.301 | ....***X | 4.301 | | 2.700 | ..X | 2.700 | | 1.499 | *.***X | 1.599 | | 1.599 | .*X | 1.600 | | 1.400 | ***X | 1.400 | | 0.000 | ********************************X | 0.000 | +-------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 06:54:07,203 | test | INFO | pool-22-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***.....****X | 1.501 | | 0.100 | X | 0.100 | | 0.000 | *******************************************************************************************************************.........*.****X | 3.001 | | 0.000 | ***********....*.****X | 1.501 | | 0.000 | ***********..............*.****X | 4.501 | | 0.300 | X | 0.300 | | 0.000 | *********************************************............*.****X | 4.099 | | 0.700 | .X | 0.700 | | 0.400 | ****..*.......*****.....*****X | 4.500 | | 0.000 | ***********....*.****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | ****....**.****..............*.****X | 5.900 | | 3.800 | ...........X | 3.800 | | 2.100 | ......****X | 2.100 | | 1.100 | ..X | 1.100 | | 0.900 | ..****X | 0.900 | | 0.000 | ***********....*.****...............*.....*****...............................................**...***X | 22.198 | | 1.799 | .....X | 1.799 | | 0.300 | ***X | 0.300 | | 0.000 | ******....*......****X | 3.100 | | 1.700 | ....******..........X | 4.499 | | 4.199 | .............****X | 4.199 | | 1.400 | ...*.***X | 1.500 | | 0.800 | .X | 0.800 | | 0.100 | X | 0.100 | | 0.000 | **********..............**.***X | 4.500 | | 1.700 | ....X | 1.700 | | 1.500 | ....*****..............*.***X | 5.999 | | 5.299 | .................X | 5.299 | | 4.602 | ..............X | 4.602 | | 4.502 | ..............***X | 4.502 | | 2.400 | .......X | 2.400 | | 0.200 | *****.............................*.***X | 8.800 | | 5.200 | ................*.****.......*...............*.****.....*....*.***X | 15.000 | | 7.300 | .......................*X | 7.300 | | 6.500 | .....................**X | 6.500 | | 6.199 | ....................*...*.****X | 7.500 | | 0.500 | X | 0.500 | | 0.000 | **********............................*..****X | 9.000 | | 6.100 | ...................X | 6.100 | | 5.900 | ...................****.....................*..****.....*......****X | 15.901 | | 11.001 | ....................................X | 11.001 | | 8.001 | ..........................X | 8.001 | | 7.501 | ........................****.....*****X | 8.901 | | 7.801 | .........................X | 7.801 | | 7.200 | .......................****.......*..........*****X | 12.299 | | 0.000 | ****************...........*....***X | 4.500 | | 4.500 | ..............*****X | 4.500 | | 1.200 | ...X | 1.200 | | 0.000 | *****....*.***X | 1.500 | | 1.100 | ..X | 1.100 | | 0.100 | X | 0.100 | | 0.000 | ****............................*...****.................*...............................................*.....***X | 29.298 | | 20.099 | ...................................................................X | 20.099 | | 12.602 | ..........................................X | 12.602 | | 11.102 | ....................................***X | 11.102 | | 9.396 | ...............................****X | 9.595 | | 7.595 | ........................X | 7.595 | | 1.600 | ....X | 1.600 | | 0.900 | ..****.*.****X | 1.500 | | 0.000 | ***************************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 06:54:07,549 | test | INFO | pool-22-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..............................*****..........................................................*............***X | 808.016 | | 784.695 | ................................................................................................X | 784.695 | | 756.205 | ............................................................................................X | 756.205 | | 734.204 | .........................................................................................***X | 734.204 | | 668.184 | .................................................................................*.****X | 677.678 | | 634.239 | .............................................................................X | 634.239 | | 600.771 | .........................................................................****..*......*..*****....*..***X | 732.438 | | 712.262 | .......................................................................................X | 712.262 | | 653.984 | ...............................................................................*****X | 653.984 | | 557.173 | ...................................................................****X | 557.173 | | 533.879 | .................................................................*X | 533.879 | | 524.450 | ...............................................................***X | 524.450 | | 500.477 | ............................................................X | 500.477 | | 499.490 | ............................................................****X | 499.490 | | 454.071 | .......................................................****X | 454.071 | | 412.216 | ..................................................X | 412.216 | | 408.125 | .................................................******.............****X | 511.380 | | 506.180 | .............................................................*........****X | 572.649 | | 449.023 | ......................................................*X | 449.023 | | 413.581 | ..................................................****.........**..***X | 507.633 | | 463.522 | ........................................................*.****.*.......*****X | 539.877 | | 516.659 | ..............................................................*****......**...***X | 582.877 | | 505.600 | .............................................................X | 505.600 | | 474.847 | .........................................................*****........*..****X | 556.770 | | 444.712 | ......................................................X | 444.712 | | 421.940 | ...................................................*****......*****..*..****...*...****X | 551.016 | | 485.720 | ...........................................................X | 485.720 | | 470.761 | .........................................................*****..*..****X | 505.668 | | 440.958 | .....................................................****X | 440.958 | | 388.423 | ...............................................*...*****........X | 477.303 | | 469.494 | .........................................................***X | 469.494 | | 466.003 | ........................................................X | 466.003 | | 395.594 | ...............................................*****...............*.****X | 518.452 | | 481.174 | ..........................................................******........****X | 543.860 | | 542.469 | ..................................................................*****X | 546.443 | | 514.139 | ..............................................................X | 514.139 | | 509.077 | ..............................................................***X | 509.077 | | 487.965 | ...........................................................X | 487.965 | | 436.393 | .....................................................****X | 436.393 | | 423.707 | ...................................................*.*****...........X | 525.038 | | 518.027 | ...............................................................****X | 518.027 | | 459.470 | .......................................................*..*****.....*.****..*X | 539.790 | | 539.469 | .................................................................***X | 539.469 | | 476.997 | ..........................................................X | 476.997 | | 466.428 | ........................................................*****..X | 483.444 | | 476.631 | .........................................................***X | 476.631 | | 470.936 | .........................................................**.***X | 477.068 | | 469.969 | .........................................................X | 469.969 | | 459.203 | .......................................................*****...*...****..X | 522.357 | | 516.759 | ..............................................................*****....X | 548.326 | | 540.939 | .................................................................***X | 540.939 | | 529.696 | ................................................................X | 529.696 | | 513.937 | ..............................................................*.***X | 520.425 | | 431.550 | ....................................................X | 431.550 | | 417.438 | ..................................................*****..........*...***X | 518.559 | | 508.874 | .............................................................X | 508.874 | | 481.109 | ..........................................................*****.*..****X | 507.176 | | 489.718 | ...........................................................X | 489.718 | | 478.221 | ..........................................................***X | 478.221 | | 473.284 | .........................................................X | 473.284 | | 469.985 | .........................................................*****.*..***X | 500.184 | | 466.852 | ........................................................X | 466.852 | | 392.702 | ...............................................****X | 392.702 | | 376.115 | .............................................**......****....X | 454.006 | | 453.276 | .......................................................****...X | 477.333 | | 471.052 | .........................................................X | 471.052 | | 457.387 | .......................................................***X | 457.387 | | 449.711 | ......................................................X | 449.711 | | 437.536 | .....................................................*****.*.....****X | 490.053 | | 438.783 | .....................................................****..X | 454.185 | | 453.858 | .......................................................X | 453.858 | | 452.667 | .......................................................***X | 452.667 | | 394.383 | ...............................................*.****.*......X | 459.664 | | 453.673 | .......................................................****X | 453.673 | | 394.985 | ...............................................*****..........*..***X | 491.169 | | 469.662 | .........................................................X | 469.662 | | 447.499 | ......................................................X | 447.499 | | 420.292 | ...................................................****.....**.....**X | 507.665 | | 489.970 | ...........................................................X | 489.970 | | 401.236 | ................................................X | 401.236 | | 366.670 | ............................................****..............X | 477.813 | | 477.713 | ..........................................................*****...****X | 503.818 | | 402.183 | ................................................X | 402.183 | | 372.167 | .............................................****...*.............*....***X | 538.020 | | 522.339 | ...............................................................X | 522.339 | | 491.617 | ...........................................................X | 491.617 | | 485.417 | ...........................................................***X | 485.417 | | 454.117 | .......................................................X | 454.117 | | 429.920 | ....................................................***X | 429.920 | | 402.717 | ................................................X | 402.717 | | 398.815 | ................................................*.***X | 409.305 | | 397.160 | ................................................X | 397.160 | | 390.561 | ...............................................****.*......*.****......**.***X | 514.104 | | 496.569 | ............................................................*.****X | 505.434 | | 487.107 | ...........................................................****....X | 519.787 | | 509.308 | ..............................................................*.***X | 520.127 | | 466.907 | ........................................................X | 466.907 | | 458.307 | .......................................................X | 458.307 | | 433.783 | ....................................................****.....*....*****.....*.****X | 551.821 | | 446.784 | ......................................................X | 446.784 | | 419.793 | ..................................................****.X | 426.384 | | 411.584 | .................................................X | 411.584 | | 398.176 | ................................................****..*.....*****...X | 479.997 | | 466.086 | ........................................................***X | 466.086 | | 452.017 | ......................................................*...****X | 469.576 | | 447.029 | ......................................................*****.......*...****X | 531.662 | | 482.011 | ..........................................................X | 482.011 | | 471.610 | .........................................................*****.......*...***X | 549.634 | | 515.120 | ..............................................................*.X | 517.631 | | 506.731 | .............................................................****.....*.****X | 556.614 | | 391.311 | ...............................................X | 391.311 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 06:54:07,838 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 69.6616666667 | 6437847040 | 4331380736 | 2414338048 / 7511588864 | 101215 / 199719 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 80.5746762446 | 6437847040 | 4242468864 | 2472681472 / 7511588864 | 100195 / 204378 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 70.9897838441 | 6437847040 | 4288966656 | 2415058944 / 7511588864 | 100400 / 203200 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 63.6937252742 | 6437847040 | 3904208896 | 3008311296 / 9530753024 | 101195 / 198856 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 62.0856319061 | 6437847040 | 4264058880 | 2464362496 / 7511588864 | 100971 / 201698 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 06:54:20,651 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 06:55:46,674 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 165019080 | 242245058 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162290608 | 271439459 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 447186104 | 444808152 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 06:55:46,677 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #20 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 06:55:56,801 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 06:55:56,801 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 06:55:56,802 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 06:56:06,967 | test | INFO | pool-22-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.28299999237 seconds to complete 2021-05-17 06:56:06,971 | test | INFO | pool-22-thread-6 | [common_lib:sleep:22] Sleep 8.28299999237 seconds. Reason: Wait after rebalance complete 2021-05-17 06:58:15,319 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0620000362396 seconds 2021-05-17 06:58:15,319 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 06:58:15,319 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 06:58:22,085 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 07:10:33,148 | test | INFO | pool-22-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******...*.....*****............................*.................*..............****.....*...*..***X | 199.706 | | 195.007 | ..........................................................................*..X | 199.868 | | 183.385 | .....................................................................****X | 183.385 | | 127.630 | ................................................X | 127.630 | | 121.120 | .............................................*****......................*..............****...*....X | 232.672 | | 215.106 | ..................................................................................***X | 215.106 | | 208.906 | ...............................................................................X | 208.906 | | 200.218 | ............................................................................*.***X | 202.589 | | 194.685 | ..........................................................................X | 194.685 | | 185.729 | ......................................................................*...****................X | 235.665 | | 230.397 | .......................................................................................****X | 230.397 | | 228.869 | .......................................................................................X | 228.869 | | 220.797 | ....................................................................................*...****..X | 234.375 | | 233.975 | .........................................................................................*****..........X | 259.014 | | 258.814 | ..................................................................................................****X | 258.814 | | 235.909 | ..........................................................................................X | 235.909 | | 227.503 | ......................................................................................***X | 227.503 | | 211.108 | ................................................................................X | 211.108 | | 161.274 | .............................................................X | 161.274 | | 158.064 | ............................................................***X | 158.064 | | 133.061 | ..................................................X | 133.061 | | 89.672 | .................................X | 89.672 | | 57.981 | .....................***X | 57.981 | | 53.784 | ...................X | 53.784 | | 52.790 | ...................X | 52.790 | | 49.386 | ..................****..**.***X | 57.484 | | 26.499 | .........X | 26.499 | | 13.700 | ....X | 13.700 | | 0.300 | ***X | 0.300 | | 0.100 | *****************X | 1.799 | | 0.900 | *X | 0.900 | | 0.600 | ***X | 0.600 | | 0.300 | X | 0.300 | | 0.200 | *******X | 1.999 | | 1.999 | **X | 1.999 | | 1.200 | X | 1.200 | | 1.200 | *****X | 1.699 | | 1.499 | ****X | 1.499 | | 0.000 | ***********X | 1.800 | | 0.000 | ***********.***********X | 3.599 | | 1.300 | ******X | 1.400 | | 1.200 | ****X | 1.200 | | 1.100 | ******..*****X | 5.898 | | 1.200 | *X | 1.200 | | 0.400 | **********X | 2.299 | | 2.200 | ***X | 2.200 | | 1.700 | X | 1.700 | | 1.366 | ******.******X | 4.299 | | 3.899 | ***X | 3.899 | | 3.199 | X | 3.199 | | 1.928 | X | 1.928 | | 1.727 | ***X | 1.727 | | 0.599 | *X | 0.599 | | 0.400 | *****************.****X | 2.597 | | 1.897 | *X | 1.897 | | 1.598 | ***X | 1.598 | | 0.498 | X | 0.498 | | 0.200 | **********X | 0.701 | | 0.700 | ******.*.***X | 6.601 | | 4.592 | X | 4.592 | | 2.293 | X | 2.293 | | 0.399 | ***********.*****X | 4.599 | | 3.697 | X | 3.697 | | 0.501 | ***********.*******.****X | 5.697 | | 2.494 | X | 2.494 | | 0.300 | *****************.*****X | 3.299 | | 1.598 | *X | 1.598 | | 0.799 | ****.******X | 4.699 | | 0.200 | ******.****X | 3.498 | | 2.198 | X | 2.198 | | 1.000 | **********X | 2.099 | | 1.998 | *****X | 2.198 | | 1.798 | X | 1.798 | | 1.399 | **X | 1.399 | +---------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:10:34,417 | test | INFO | pool-22-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.*.****X | 4.694 | | 2.898 | ******........................*......*****......*...****X | 77.265 | | 71.759 | ....................................X | 71.759 | | 65.065 | .................................*****....*.****.*.*....***X | 86.176 | | 84.562 | ...........................................X | 84.562 | | 84.454 | ...........................................X | 84.454 | | 81.966 | .........................................****..*.X | 87.770 | | 83.860 | ..........................................****.*..*...*****.....*...*****.X | 112.651 | | 108.247 | .......................................................***X | 108.247 | | 103.387 | .....................................................*.X | 106.893 | | 104.392 | .....................................................****.....*...****X | 119.568 | | 118.757 | .............................................................X | 118.757 | | 113.554 | ..........................................................***X | 113.554 | | 103.057 | ....................................................*..........*......*****..........*.***X | 153.850 | | 128.960 | ..................................................................X | 128.960 | | 117.282 | ............................................................*X | 117.282 | | 101.172 | ...................................................*X | 101.172 | | 101.172 | ...................................................*......*..*.....****....*........*.................****......X | 191.363 | | 170.667 | ........................................................................................X | 170.667 | | 162.163 | ...................................................................................****X | 162.163 | | 153.650 | ...............................................................................X | 153.650 | | 152.850 | ..............................................................................*****....*..*****X | 164.054 | | 156.648 | ................................................................................****X | 156.648 | | 150.766 | .............................................................................*......****......*.*****.X | 177.466 | | 170.361 | ........................................................................................****X | 170.361 | | 151.167 | .............................................................................*****.X | 151.744 | | 141.876 | .........................................................................X | 141.876 | | 141.876 | .........................................................................**X | 141.876 | | 141.876 | .........................................................................X | 141.876 | | 132.380 | ....................................................................X | 132.380 | | 103.298 | ....................................................*****.X | 103.598 | | 96.283 | .................................................****X | 96.283 | | 78.685 | ........................................X | 78.685 | | 41.791 | ....................****X | 41.791 | | 30.390 | ..............X | 30.390 | | 1.800 | ****.*****X | 2.099 | | 1.999 | *X | 1.999 | | 1.900 | ****.*****X | 2.799 | | 2.000 | *X | 2.000 | | 1.800 | ****.******X | 3.399 | | 2.699 | **********X | 3.499 | | 2.000 | X | 2.000 | | 1.500 | ****X | 1.500 | | 0.700 | ******.*.****X | 4.299 | | 3.600 | X | 3.600 | | 3.199 | ****X | 3.199 | | 1.948 | *X | 1.948 | | 1.547 | **X | 1.547 | | 0.898 | ************.***********X | 2.497 | | 0.897 | *X | 0.897 | | 0.400 | **************X | 1.000 | | 0.800 | *X | 0.800 | | 0.700 | *****.****X | 3.399 | | 1.496 | X | 1.496 | | 0.500 | ************.*.****X | 4.396 | | 3.297 | X | 3.297 | | 1.303 | ***X | 1.303 | | 0.801 | X | 0.801 | | 0.701 | ****X | 1.399 | | 1.399 | *X | 1.498 | | 1.200 | **********X | 1.799 | | 1.199 | X | 1.199 | | 0.900 | **********X | 1.300 | | 1.200 | ******.*****X | 2.399 | | 1.799 | ******.X | 2.499 | | 2.100 | *****X | 2.100 | | 0.200 | ************************.****X | 2.099 | | 1.999 | X | 1.999 | | 1.299 | ***X | 1.299 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:10:35,108 | test | INFO | pool-22-thread-27 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...*.........*....***X | 226.866 | | 84.706 | .....*X | 84.706 | | 33.493 | .****......................................................................**............................***X | 1367.004 | | 802.813 | .........................................................X | 802.813 | | 802.813 | .........................................................X | 802.813 | | 500.317 | ...................................**X | 500.317 | | 500.317 | ...................................X | 500.317 | | 451.758 | ................................*****X | 461.836 | | 421.747 | .............................X | 421.747 | | 419.001 | .............................*X | 419.001 | | 419.001 | .............................***.............*..****X | 622.486 | | 588.546 | ..........................................*****X | 589.334 | | 506.826 | ....................................X | 506.826 | | 490.743 | ..................................*****.X | 496.466 | | 489.973 | ..................................****X | 489.973 | | 381.684 | ..........................X | 381.684 | | 334.247 | .......................*****...*......*****.........*....****X | 637.888 | | 361.021 | .........................X | 361.021 | | 289.407 | ....................***X | 289.407 | | 274.008 | ...................X | 274.008 | | 244.116 | ................X | 244.116 | | 232.108 | ...............***X | 232.108 | | 232.014 | ...............*..*..****......*..****X | 390.446 | | 353.338 | ........................X | 353.338 | | 326.508 | ......................****X | 326.508 | | 300.539 | ....................X | 300.539 | | 292.323 | ....................*****..**...****............**.***X | 543.737 | | 424.275 | ..............................*X | 424.275 | | 384.475 | ...........................****...........*....****X | 587.944 | | 404.500 | ............................X | 404.500 | | 358.080 | .........................**X | 358.080 | | 358.080 | .........................**..*.****......*****X | 483.523 | | 410.122 | .............................X | 410.122 | | 386.608 | ...........................X | 386.608 | | 386.608 | ...........................**X | 386.608 | | 386.608 | ...........................X | 386.608 | | 359.817 | .........................*.*****..X | 399.011 | | 380.007 | ..........................*****.....*...****X | 488.088 | | 484.091 | ..................................*X | 484.091 | | 473.815 | .................................****..*...*****................*...X | 803.505 | | 803.505 | .........................................................**X | 803.505 | | 803.505 | .........................................................*.X | 810.296 | | 801.157 | .........................................................***X | 801.157 | | 787.449 | ........................................................**..X | 810.212 | | 810.212 | ..........................................................***...*X | 859.958 | | 854.215 | .............................................................***X | 854.215 | | 605.836 | ...........................................*X | 605.836 | | 546.930 | .......................................****.....*.**X | 641.959 | | 641.959 | .............................................**.*X | 642.667 | | 626.550 | ............................................***X | 626.550 | | 573.843 | ........................................*.*****...X | 620.730 | | 592.259 | ..........................................****.*.*****.X | 631.146 | | 600.568 | ..........................................****X | 600.568 | | 562.456 | ........................................X | 562.456 | | 559.386 | .......................................****X | 559.386 | | 500.697 | ...................................*X | 500.697 | | 475.957 | .................................****..*...*****..X | 565.081 | | 526.263 | .....................................****X | 526.263 | | 508.717 | ....................................*.****..X | 553.927 | | 550.833 | .......................................*.***X | 566.213 | | 563.200 | ........................................*..X | 595.254 | | 572.558 | ........................................***X | 572.558 | | 534.277 | ......................................X | 534.277 | | 462.491 | ................................X | 462.491 | | 442.391 | ...............................****.*.*..****..X | 532.216 | | 526.934 | .....................................****X | 526.934 | | 486.939 | ..................................X | 486.939 | | 479.875 | ..................................*****....*..****X | 567.471 | | 558.787 | .......................................*****X | 560.378 | | 490.314 | ..................................X | 490.314 | | 454.226 | ................................*****.......**..***X | 584.863 | | 548.329 | .......................................X | 548.329 | | 515.762 | ....................................******.****X | 530.434 | | 514.320 | ....................................X | 514.320 | | 506.159 | ....................................*****..*..X | 572.307 | | 570.339 | ........................................***X | 570.339 | | 531.862 | .....................................X | 531.862 | | 524.326 | .....................................******..*****X | 550.076 | | 535.440 | ......................................*****X | 535.841 | | 533.681 | ......................................****X | 533.681 | | 517.596 | ....................................X | 517.596 | | 498.996 | ...................................*..*****......X | 604.447 | | 592.000 | ..........................................****X | 592.000 | | 582.729 | .........................................*..*****X | 603.739 | | 572.378 | ........................................****X | 572.378 | | 551.196 | .......................................*.*****............******......****X | 816.739 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 07:10:35,387 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 07:10:35,434 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 07:10:45,217 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 07:10:45,417 | test | INFO | pool-22-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.521657972367 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 42.264037734 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 70.3645060751 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 30.3128281195 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 21.2856165184 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 07:10:45,461 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 07:10:50,601 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 07:10:52,905 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 07:10:55,747 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.01 2021-05-17 07:11:00,861 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 4.67 2021-05-17 07:11:05,947 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 7.65 2021-05-17 07:11:11,066 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 9.87 2021-05-17 07:11:16,138 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 12.96 2021-05-17 07:11:21,217 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 17.4 2021-05-17 07:11:26,313 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 21.03 2021-05-17 07:11:31,391 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 24.45 2021-05-17 07:11:36,364 | test | INFO | pool-22-thread-29 | [table_view:display:72] Keys failed in bucket1:HyNGMRr3ZuFhkDyMI%3-42-471000:jFwH-42-473000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:11:36,484 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 27.97 2021-05-17 07:11:41,552 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 31.38 2021-05-17 07:11:45,546 | test | INFO | pool-22-thread-12 | [table_view:display:72] Keys failed in bucket1:w2gQN-51-219000:f5InZUfeYOAjZjr9Tp-51-347000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:11:45,546 | test | INFO | pool-22-thread-12 | [table_view:display:72] Keys failed in bucket1:w2gQN-51-219000:f5InZUfeYOAjZjr9Tp-51-347000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-568 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:11:46,625 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 07:11:51,736 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 35.73 2021-05-17 07:11:56,813 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 38.72 2021-05-17 07:12:00,628 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in bucket2:mVQ1B8cdp7Ds--19-279000:e2Xlf5mzWTIlP-19-404000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:00,628 | test | INFO | pool-22-thread-14 | [table_view:display:72] Keys failed in bucket2:UsY-19-603000:qr-19-716000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:00,630 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 07:12:01,894 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 40.7 2021-05-17 07:12:06,967 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 42.72 2021-05-17 07:12:07,821 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in bucket2:mVQ1B8cdp7Ds--19-279000:e2Xlf5mzWTIlP-19-404000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-936 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-950 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:11,315 | test | INFO | pool-22-thread-4 | [table_view:display:72] Keys failed in bucket2:J-50-71000:bhRjfqtBD%tE-50-179000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:11,315 | test | INFO | pool-22-thread-4 | [table_view:display:72] Keys failed in bucket2:J-50-71000:bhRjfqtBD%tE-50-179000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-96 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:12,085 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 45.65 2021-05-17 07:12:17,186 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 48.16 2021-05-17 07:12:22,283 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 51.95 2021-05-17 07:12:27,367 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 55.53 2021-05-17 07:12:32,453 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 59.16 2021-05-17 07:12:37,561 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 62.96 2021-05-17 07:12:42,665 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.89 2021-05-17 07:12:43,661 | test | INFO | pool-22-thread-19 | [table_view:display:72] Keys failed in bucket2:_default:d-JNe0%b_Xp4-52-541000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,663 | test | INFO | pool-22-thread-11 | [table_view:display:72] Keys failed in bucket2:VJ8HTV%2eeR6724LLF6-49-207000:7mHULQQF0TpwanqG3u7-49-428000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,663 | test | INFO | pool-22-thread-19 | [table_view:display:72] Keys failed in bucket2:_default:d-JNe0%b_Xp4-52-541000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,661 | test | INFO | pool-22-thread-21 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:T-49-852000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,664 | test | INFO | pool-22-thread-11 | [table_view:display:72] Keys failed in bucket2:VJ8HTV%2eeR6724LLF6-49-207000:7mHULQQF0TpwanqG3u7-49-428000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,664 | test | INFO | pool-22-thread-8 | [table_view:display:72] Keys failed in bucket2:J-50-71000:f%%tq4uxAv4P-50-179000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,664 | test | INFO | pool-22-thread-21 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:T-49-852000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,664 | test | INFO | pool-22-thread-8 | [table_view:display:72] Keys failed in bucket2:J-50-71000:f%%tq4uxAv4P-50-179000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,664 | test | INFO | pool-22-thread-1 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:vjwsBdkHXDy-49-853000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:43,665 | test | INFO | pool-22-thread-1 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:vjwsBdkHXDy-49-853000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1672 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:46,153 | test | INFO | pool-22-thread-15 | [table_view:display:72] Keys failed in bucket2:UsY-19-603000:qr-19-716000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-28 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-169 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-120 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:47,736 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 67.21 2021-05-17 07:12:51,767 | test | INFO | pool-22-thread-2 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-638 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:51,828 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in default:scope1:collections_2 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1119 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1089 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:52,834 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 69.54 2021-05-17 07:12:54,280 | test | INFO | pool-22-thread-19 | [table_view:display:72] Keys failed in bucket2:_default:d-JNe0%b_Xp4-52-541000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1889 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,280 | test | INFO | pool-22-thread-11 | [table_view:display:72] Keys failed in bucket2:VJ8HTV%2eeR6724LLF6-49-207000:7mHULQQF0TpwanqG3u7-49-428000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1889 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-21 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:T-49-852000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1889 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-1 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:vjwsBdkHXDy-49-853000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1889 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-8 | [table_view:display:72] Keys failed in bucket2:J-50-71000:f%%tq4uxAv4P-50-179000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1889 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-11 | [table_view:display:72] Keys failed in bucket2:VJ8HTV%2eeR6724LLF6-49-207000:7mHULQQF0TpwanqG3u7-49-428000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-8 | [table_view:display:72] Keys failed in bucket2:J-50-71000:f%%tq4uxAv4P-50-179000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,283 | test | INFO | pool-22-thread-21 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:T-49-852000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,282 | test | INFO | pool-22-thread-19 | [table_view:display:72] Keys failed in bucket2:_default:d-JNe0%b_Xp4-52-541000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:54,283 | test | INFO | pool-22-thread-1 | [table_view:display:72] Keys failed in bucket2:G9_dHSfJq8MN%XS-49-719000:vjwsBdkHXDy-49-853000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1889 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:55,997 | test | INFO | pool-22-thread-10 | [table_view:display:72] Keys failed in default:scope2:1XB2iUVdVou-42-479000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:12:57,921 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 72.63 2021-05-17 07:13:00,167 | test | INFO | pool-22-thread-23 | [table_view:display:72] Keys failed in default:scope1:zF0-42-477000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1326 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1205 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1370 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1363 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:00,397 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in default:_default:oGdUcQaM-42-475000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-929 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-936 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-933 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-967 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-976 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:00,640 | test | INFO | pool-22-thread-30 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:-L%eaCoIOm4-42-485000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1411 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1402 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1395 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:02,688 | test | INFO | pool-22-thread-17 | [table_view:display:72] Keys failed in default:4yNQhJ9G%3T_mgMzCoMP-51-876000:qnUnPKQu9KS6FRLVU-51-987000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:02,690 | test | INFO | pool-22-thread-17 | [table_view:display:72] Keys failed in default:4yNQhJ9G%3T_mgMzCoMP-51-876000:qnUnPKQu9KS6FRLVU-51-987000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-740 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:03,009 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 75.44 2021-05-17 07:13:03,223 | test | INFO | pool-22-thread-14 | [table_view:display:72] Keys failed in default:UD6owGuXFA-21-18000:I-p-21-129000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-603 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-624 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-743 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-750 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:04,734 | test | INFO | pool-22-thread-18 | [table_view:display:72] Keys failed in default:scope1:SB5X1DOr2xL-42-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-539 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-603 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-624 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-621 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-664 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:05,680 | test | INFO | pool-22-thread-2 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-819 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-729 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-833 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-796 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-873 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-750 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:05,766 | test | INFO | pool-22-thread-20 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:uNAIV_gP%Qw0-42-483000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1071 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1050 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-989 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1173 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1076 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1003 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:08,102 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 77.77 2021-05-17 07:13:09,369 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in default:scope1:collections_2 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1279 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1205 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1239 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1363 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1253 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:09,437 | test | INFO | pool-22-thread-10 | [table_view:display:72] Keys failed in default:scope2:1XB2iUVdVou-42-479000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1395 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1388 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:12,515 | test | INFO | pool-22-thread-22 | [table_view:display:72] Keys failed in default:T-51-504000:E0QGUnKYDXwoR5iGJ-51-708000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:12,516 | test | INFO | pool-22-thread-22 | [table_view:display:72] Keys failed in default:T-51-504000:E0QGUnKYDXwoR5iGJ-51-708000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:12,592 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in default:_default:bwY-42-476000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1136 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1005 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1040 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1065 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1033 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1134 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1002 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:13,180 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 80.43 2021-05-17 07:13:13,500 | test | INFO | pool-22-thread-23 | [table_view:display:72] Keys failed in default:scope1:zF0-42-477000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1386 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1388 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:13,523 | test | INFO | pool-22-thread-26 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:u%D_OqSAXI-42-484000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1386 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1388 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:16,216 | test | INFO | pool-22-thread-17 | [table_view:display:72] Keys failed in default:4yNQhJ9G%3T_mgMzCoMP-51-876000:qnUnPKQu9KS6FRLVU-51-987000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:16,216 | test | INFO | pool-22-thread-17 | [table_view:display:72] Keys failed in default:4yNQhJ9G%3T_mgMzCoMP-51-876000:qnUnPKQu9KS6FRLVU-51-987000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-858 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:17,688 | test | INFO | pool-22-thread-2 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-915 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-942 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:18,303 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 82.87 2021-05-17 07:13:19,059 | test | INFO | pool-22-thread-18 | [table_view:display:72] Keys failed in default:scope1:SB5X1DOr2xL-42-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-713 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-810 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-734 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-710 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-773 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-851 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:19,061 | test | INFO | pool-22-thread-14 | [table_view:display:72] Keys failed in default:UD6owGuXFA-21-18000:I-p-21-129000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-915 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-810 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-942 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-851 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-980 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:21,289 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in default:scope1:collections_2 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1432 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:21,934 | test | INFO | pool-22-thread-20 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:uNAIV_gP%Qw0-42-483000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1225 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1305 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1229 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1295 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1340 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1264 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:23,426 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 85.63 2021-05-17 07:13:26,816 | test | INFO | pool-22-thread-10 | [table_view:display:72] Keys failed in default:_default:ub4p8k-42-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-649 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-504 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-615 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-623 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-677 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-556 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-545 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-652 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-598 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-595 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:28,421 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in default:_default:bwY-42-476000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1225 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1305 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1229 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1318 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1261 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1295 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1264 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1198 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1221 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1288 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:28,515 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 88.13 2021-05-17 07:13:29,348 | test | INFO | pool-22-thread-26 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:uYqhmbMr-42-484000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1082 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-994 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1137 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1073 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1176 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1156 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1112 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:33,138 | test | INFO | pool-22-thread-18 | [table_view:display:72] Keys failed in default:scope1:SB5X1DOr2xL-42-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-908 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-971 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:33,592 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 90.84 2021-05-17 07:13:35,309 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:usGy5YX7o8-42-482000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1048 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1137 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1049 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1152 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1132 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1165 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1014 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:35,769 | test | INFO | pool-22-thread-20 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:uNAIV_gP%Qw0-42-483000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1415 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1417 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:38,677 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 93.6 2021-05-17 07:13:40,542 | test | INFO | pool-22-thread-10 | [table_view:display:72] Keys failed in default:_default:ub4p8k-42-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-803 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-815 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-835 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-769 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-711 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-744 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-865 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-784 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-881 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-893 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-860 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:41,391 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in default:_default:bwY-42-476000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1433 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1415 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1426 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:42,897 | test | INFO | pool-22-thread-15 | [table_view:display:72] Keys failed in default:_default:G2nunngeY9_a%BmpMOJT-42-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-614 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-538 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-513 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-522 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-500 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-567 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-512 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-501 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-681 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-660 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:43,749 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 96.2 2021-05-17 07:13:44,028 | test | INFO | pool-22-thread-26 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:uYqhmbMr-42-484000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1367 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1247 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1369 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1348 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1304 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1349 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1294 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1196 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1187 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1199 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1354 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:48,528 | test | INFO | pool-22-thread-23 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:-L%eaCoIOm4-42-485000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-619 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-636 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-538 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-568 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-500 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-685 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:48,911 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: running, progress: 98.81 2021-05-17 07:13:51,242 | test | INFO | pool-22-thread-9 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:usGy5YX7o8-42-482000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1378 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1369 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1327 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1228 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1271 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1240 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1362 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1263 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1354 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1377 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1234 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1245 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:51,484 | test | INFO | pool-22-thread-20 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:PDil-42-486000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1170 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1114 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1159 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1128 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1029 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1108 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1084 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1020 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1075 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1043 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1098 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1133 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1100 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:54,293 | test | INFO | pool-22-thread-10 | [table_view:display:72] Keys failed in default:_default:ub4p8k-42-474000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-966 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-978 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:56,194 | test | INFO | pool-22-thread-28 | [table_view:display:72] Keys failed in default:pRBhm9zlg_-42-481000:doeEoQtuCpmJx2-42-483000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1170 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1108 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1030 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1041 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1098 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1100 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1145 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1057 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:56,233 | test | INFO | pool-22-thread-15 | [table_view:display:72] Keys failed in default:_default:G2nunngeY9_a%BmpMOJT-42-476000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-708 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-715 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-716 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-714 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-701 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-874 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-886 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-762 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-862 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-752 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:58,493 | test | INFO | pool-22-thread-26 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:uYqhmbMr-42-484000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1390 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1421 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:13:59,035 | test | INFO | pool-22-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 07:13:59,141 | test | INFO | pool-22-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 193.724999905 sec 2021-05-17 07:13:59,997 | test | INFO | pool-22-thread-23 | [table_view:display:72] Keys failed in default:k%qJV6Btrv-w-42-484000:-L%eaCoIOm4-42-485000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-889 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-874 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-797 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-831 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:14:03,134 | test | INFO | pool-22-thread-18 | [table_view:display:72] Keys failed in default:scope2:1XB2iUVdVou-42-479000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 07:15:23,069 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 07:15:23,069 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 7 docs from 0 to 1000 2021-05-17 07:15:25,569 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 07:15:25,571 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 07:15:25,576 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 07:15:31,624 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 07:15:31,625 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 07:15:31,625 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 07:15:31,627 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 07:17:40,812 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 07:17:40,815 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 07:17:41,865 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 07:17:41,865 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 20 docs from 0 to 1000 2021-05-17 07:17:41,867 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-17 07:17:41,867 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 4 docs from 980 to 1440 2021-05-17 07:17:41,869 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 07:17:41,869 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 9 docs from 980 to 1440 2021-05-17 07:17:41,869 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 07:17:41,871 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 5 docs from 0 to 1000 2021-05-17 07:17:41,871 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 18 docs from 980 to 1440 2021-05-17 07:17:41,871 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-17 07:17:41,871 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 25 docs from 500 to 980 2021-05-17 07:17:41,872 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 28 docs from 980 to 1440 2021-05-17 07:17:41,874 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 36 docs from 980 to 1440 2021-05-17 07:17:41,875 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 34 docs from 500 to 980 2021-05-17 07:17:41,875 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 28 docs from 980 to 1440 2021-05-17 07:17:41,877 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 26 docs from 980 to 1440 2021-05-17 07:17:41,878 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 24 docs from 500 to 980 2021-05-17 07:17:41,878 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-17 07:17:41,880 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 18 docs from 980 to 1440 2021-05-17 07:17:41,881 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 11 docs from 980 to 1440 2021-05-17 07:17:41,881 | test | ERROR | pool-22-thread-6 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 07:27:38,036 | test | INFO | pool-22-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******X | 0.200 | | 0.100 | *********.........*..****.*...****....................................................*...................***X | 122.211 | | 35.000 | .......................****X | 35.000 | | 30.603 | ....................X | 30.603 | | 27.892 | ..................****........*....****.........................****....................................X | 130.826 | | 125.764 | .......................................................................................****X | 125.764 | | 55.106 | .....................................*.......****...........X | 80.304 | | 78.805 | ......................................................***X | 78.805 | | 65.792 | .............................................***X | 65.792 | | 56.189 | ......................................X | 56.189 | | 55.203 | .....................................***X | 55.203 | | 48.710 | .................................X | 48.710 | | 47.808 | ................................***....X | 53.775 | | 52.173 | ...................................*****..............*......**X | 79.951 | | 72.729 | ..................................................X | 72.729 | | 63.098 | ...........................................X | 63.098 | | 57.096 | .......................................****................*...*...****.*....****X | 94.899 | | 92.101 | ................................................................X | 92.101 | | 71.904 | .................................................*****................................*.................*****.X | 141.597 | | 115.719 | ................................................................................****X | 115.719 | | 62.001 | ..........................................X | 62.001 | | 56.001 | ......................................***X | 56.001 | | 54.603 | .....................................*.*..****....*.....*.....****......*.X | 88.107 | | 85.104 | ...........................................................***X | 85.104 | | 74.088 | ...................................................*X | 74.088 | | 71.795 | .................................................***X | 71.795 | | 64.717 | ............................................*X | 64.717 | | 61.415 | ..........................................***X | 61.415 | | 55.192 | .....................................**.****...******.....*...*****.X | 72.305 | | 70.408 | ................................................*****.......*..****X | 82.803 | | 58.202 | ........................................X | 58.202 | | 54.103 | .....................................****X | 54.103 | | 53.497 | ....................................******....X | 59.197 | | 57.197 | .......................................****X | 57.197 | | 52.104 | ...................................*.*****X | 53.798 | | 49.500 | .................................***X | 49.500 | | 37.601 | .........................X | 37.601 | | 10.696 | ......X | 10.696 | | 0.400 | ****..........*.....................*...........*****........*....****.......*..............*........***X | 117.599 | | 116.487 | .................................................................................*...X | 120.687 | | 118.088 | ..................................................................................***X | 118.088 | | 105.083 | .........................................................................X | 105.083 | | 99.103 | ....................................................................X | 99.103 | | 94.405 | .................................................................***X | 94.405 | | 66.197 | .............................................X | 66.197 | | 59.706 | .........................................*****.*X | 60.904 | | 59.201 | ........................................****.............*....*****............*...........****X | 115.008 | | 101.907 | ......................................................................*...****X | 105.104 | | 101.687 | ......................................................................X | 101.687 | | 91.191 | ...............................................................****X | 91.191 | | 61.505 | ..........................................X | 61.505 | | 54.708 | .....................................*****..**.***X | 58.901 | | 32.098 | .....................X | 32.098 | | 18.100 | ...........****X | 18.100 | | 10.101 | ......X | 10.101 | | 2.900 | .*****.....................*..........*****........X | 59.203 | | 55.404 | ......................................****X | 55.404 | | 10.699 | ......X | 10.699 | | 0.200 | ****X | 0.200 | | 0.100 | ***********X | 0.600 | | 0.300 | X | 0.300 | | 0.000 | **********.........*................*..........****..X | 52.680 | | 50.982 | ...................................****X | 50.982 | | 40.510 | ...........................X | 40.510 | | 12.305 | .......*****.........................*........****X | 59.106 | | 26.704 | .................X | 26.704 | | 11.201 | ......****X | 11.201 | | 0.800 | *X | 0.800 | | 0.400 | **********X | 0.900 | | 0.600 | *******.*****X | 2.400 | | 1.800 | *****............*....*****.................*......***X | 57.514 | | 51.514 | ...................................X | 51.514 | | 32.712 | ......................X | 32.712 | | 23.098 | ...............***X | 23.098 | | 16.297 | ..........X | 16.297 | | 5.600 | ..X | 5.600 | | 1.300 | *****.X | 2.300 | | 2.300 | ****X | 2.300 | | 0.900 | X | 0.900 | | 0.600 | ******.****X | 1.600 | | 1.300 | X | 1.300 | | 0.999 | *****.***********.X | 2.900 | | 2.399 | X | 2.399 | | 2.200 | ***X | 2.200 | | 1.900 | X | 1.900 | | 1.500 | X | 1.500 | | 1.000 | ***X | 1.000 | | 0.800 | X | 0.800 | | 0.500 | X | 0.500 | | 0.200 | ***X | 0.200 | | 0.200 | *******......*......*****..............................*....***X | 65.997 | | 34.302 | .......................X | 34.302 | | 1.100 | *****.*.****X | 3.300 | | 2.800 | X | 2.800 | | 2.500 | ****X | 2.500 | | 1.200 | *..*****............................*..............*..........****X | 77.217 | | 19.602 | ............X | 19.602 | | 3.000 | .***X | 3.000 | | 2.900 | .******X | 3.401 | | 2.300 | X | 2.300 | | 1.200 | ****.*****X | 2.300 | | 1.600 | X | 1.600 | | 1.500 | *****X | 2.000 | | 1.600 | ****X | 1.600 | | 0.700 | X | 0.700 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:27:38,730 | test | INFO | pool-22-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...............................................****X | 135.209 | | 92.232 | ...............................*****...............****....*...................****......................*........***X | 286.461 | | 264.554 | ...........................................................................................X | 264.554 | | 167.238 | .........................................................*****.......................****X | 233.005 | | 89.855 | ..............................X | 89.855 | | 35.126 | ...........****.........*.....................****..****.X | 129.032 | | 126.046 | ...........................................****..*...***X | 140.904 | | 136.500 | ..............................................*****............***X | 169.879 | | 169.781 | ..........................................................****X | 170.616 | | 158.536 | ......................................................X | 158.536 | | 156.929 | .....................................................***.*......*.***X | 180.015 | | 131.746 | ............................................X | 131.746 | | 117.675 | ........................................****......*..*****.......X | 162.596 | | 161.393 | .......................................................****.....X | 176.996 | | 157.193 | .....................................................X | 157.193 | | 156.805 | .....................................................****.....X | 169.908 | | 153.100 | ....................................................*****.....X | 167.998 | | 159.704 | ......................................................*****.........*....****X | 195.405 | | 147.398 | ..................................................X | 147.398 | | 139.095 | ...............................................*****...........*..****X | 177.184 | | 150.499 | ...................................................X | 150.499 | | 144.814 | .................................................*****...X | 152.811 | | 142.208 | ................................................****X | 142.208 | | 88.183 | .............................X | 88.183 | | 78.294 | ..........................*****......*..***X | 100.722 | | 90.686 | ..............................*X | 90.686 | | 88.387 | .............................****.X | 91.012 | | 90.612 | ..............................****X | 90.612 | | 64.695 | .....................X | 64.695 | | 57.200 | ..................****.*....*.***X | 72.209 | | 66.403 | ......................X | 66.403 | | 62.903 | ....................*.****.X | 66.104 | | 64.604 | .....................****X | 64.604 | | 60.497 | ....................*****X | 60.499 | | 57.697 | ...................*X | 57.697 | | 56.097 | ..................***X | 56.097 | | 55.408 | ..................*X | 55.408 | | 54.706 | ..................***X | 54.706 | | 17.195 | .....X | 17.195 | | 3.700 | ****X | 3.700 | | 0.500 | *****X | 0.600 | | 0.400 | X | 0.400 | | 0.400 | **********X | 1.600 | | 1.200 | *X | 1.200 | | 1.000 | ***X | 1.000 | | 0.200 | X | 0.200 | | 0.000 | *****.......*....*****..*.........*****.X | 66.305 | | 62.303 | ....................****X | 62.303 | | 30.497 | .........X | 30.497 | | 22.899 | ......****X | 22.899 | | 5.500 | X | 5.500 | | 0.700 | **********...*...*..****..........**..***X | 59.809 | | 58.004 | ...................X | 58.004 | | 54.104 | .................X | 54.104 | | 51.701 | .................***X | 51.701 | | 50.501 | ................*....*.***X | 63.998 | | 32.697 | ..........X | 32.697 | | 15.499 | ....X | 15.499 | | 1.000 | ***X | 1.000 | | 0.800 | X | 0.800 | | 0.400 | **********X | 1.500 | | 1.100 | X | 1.100 | | 0.500 | X | 0.500 | | 0.200 | *****...................*........****X | 77.706 | | 25.402 | .......X | 25.402 | | 4.502 | ****X | 4.502 | | 0.200 | X | 0.200 | | 0.000 | **********X | 1.400 | | 0.600 | ****X | 0.600 | | 0.100 | X | 0.100 | | 0.000 | *******************.****X | 3.000 | | 1.400 | X | 1.400 | | 0.400 | ****************X | 1.200 | | 1.000 | *X | 1.000 | | 0.900 | ***X | 0.900 | | 0.700 | X | 0.700 | | 0.300 | *****..........*....**********X | 41.410 | | 10.110 | ..X | 10.110 | | 0.200 | ****************X | 1.800 | | 1.400 | X | 1.400 | | 0.500 | *****X | 1.300 | | 1.300 | ****X | 1.300 | | 0.800 | X | 0.800 | | 0.500 | **********X | 2.100 | | 1.000 | *X | 1.000 | | 0.500 | ***X | 0.500 | | 0.500 | X | 0.500 | | 0.300 | ****X | 0.300 | | 0.100 | *****.*...**..***.....*.......*....***X | 63.209 | | 45.807 | ..............X | 45.807 | | 16.402 | ....X | 16.402 | | 1.400 | ****X | 1.400 | | 0.500 | X | 0.500 | | 0.000 | ****************..*....*..****.....*...........*..***X | 76.593 | | 57.191 | ..................X | 57.191 | | 16.802 | ....X | 16.802 | | 0.700 | ***X | 0.700 | | 0.600 | ************X | 2.000 | | 1.700 | X | 1.700 | | 1.300 | ****...*........**.....***X | 47.172 | | 44.704 | ..............*X | 44.704 | | 35.902 | ...........**X | 35.902 | | 25.601 | .......X | 25.601 | | 7.100 | .X | 7.100 | | 1.300 | ****X | 1.300 | | 0.700 | X | 0.700 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:27:40,138 | test | INFO | pool-22-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...................................................................****X | 443.246 | | 285.398 | ..........................................*****............................***X | 468.223 | | 281.195 | .........................................X | 281.195 | | 237.218 | ...................................***X | 237.218 | | 203.201 | .............................X | 203.201 | | 175.833 | .........................****.................................*............X | 466.962 | | 466.962 | ......................................................................***............*.....***X | 579.472 | | 429.106 | ................................................................X | 429.106 | | 120.244 | .................****.........*...........****.........*X | 310.426 | | 310.426 | ..............................................**...*....****.X | 365.524 | | 333.833 | .................................................***X | 333.833 | | 223.596 | .................................X | 223.596 | | 217.102 | ................................****..............***X | 314.256 | | 232.395 | ..................................X | 232.395 | | 207.230 | ..............................****.........................*.........***X | 430.388 | | 304.659 | .............................................*.****.............X | 397.683 | | 327.568 | ................................................***X | 327.568 | | 136.401 | ...................X | 136.401 | | 127.194 | ..................*****....................*.......****.X | 311.184 | | 295.778 | ............................................X | 295.778 | | 288.205 | ..........................................****.*.*****...............*......****X | 436.382 | | 384.280 | .........................................................X | 384.280 | | 346.322 | ...................................................X | 346.322 | | 346.322 | ...................................................**X | 346.322 | | 346.322 | ...................................................X | 346.322 | | 231.801 | ..................................*X | 231.801 | | 209.601 | ..............................****.......*.X | 262.678 | | 262.678 | ......................................****.........................*...........****X | 497.450 | | 466.079 | .....................................................................X | 466.079 | | 414.209 | ..............................................................****X | 414.209 | | 392.913 | ..........................................................X | 392.913 | | 288.581 | ..........................................****X | 288.581 | | 248.363 | ....................................X | 248.363 | | 227.349 | .................................****X | 227.349 | | 200.073 | .............................X | 200.073 | | 181.980 | ..........................******.*****.........*...***X | 266.300 | | 235.483 | ..................................X | 235.483 | | 195.307 | ............................X | 195.307 | | 161.620 | .......................****.X | 168.110 | | 162.610 | .......................*..***X | 176.702 | | 163.402 | .......................*.*****......******...................*.......*****...*..****X | 408.740 | | 406.686 | ............................................................X | 406.686 | | 394.691 | ...........................................................*****..............*....****X | 518.891 | | 425.007 | ...............................................................*X | 425.007 | | 382.999 | .........................................................***X | 382.999 | | 346.419 | ...................................................*....*****............*....****X | 477.399 | | 424.051 | ...............................................................X | 424.051 | | 399.061 | ...........................................................****X | 399.061 | | 391.825 | ..........................................................*..*****.......*..****X | 465.946 | | 374.026 | .......................................................X | 374.026 | | 340.608 | ..................................................****X | 340.608 | | 326.282 | ................................................X | 326.282 | | 321.492 | ...............................................****X | 321.492 | | 267.427 | .......................................X | 267.427 | | 261.120 | ......................................*****..*..****..*****X | 300.469 | | 208.516 | ..............................X | 208.516 | | 176.827 | .........................*****......*....*****.....*..*****........................**........***X | 496.724 | | 433.375 | ................................................................X | 433.375 | | 412.465 | .............................................................****X | 412.465 | | 320.009 | ...............................................*X | 320.009 | | 290.112 | ...........................................****...................**....***X | 445.397 | | 329.698 | .................................................X | 329.698 | | 302.519 | .............................................*****.....*.................****X | 450.611 | | 393.507 | ..........................................................X | 393.507 | | 355.827 | .....................................................*****........*...****X | 429.158 | | 404.198 | ............................................................X | 404.198 | | 388.820 | ..........................................................****X | 388.820 | | 247.820 | ....................................*X | 247.820 | | 230.236 | ..................................****........................**...***X | 409.227 | | 348.876 | ....................................................X | 348.876 | | 210.219 | ..............................*****.......................*.........*****......X | 457.013 | | 437.713 | .................................................................****X | 437.713 | | 369.191 | .......................................................*.******X | 376.745 | | 372.245 | .......................................................****.....******....X | 427.460 | | 417.179 | ..............................................................****X | 417.179 | | 353.698 | ....................................................X | 353.698 | | 331.435 | .................................................****X | 331.435 | | 192.826 | ............................*X | 192.826 | | 140.128 | ....................****.........*.....*........****..............*.......*......****......*..*..***X | 529.297 | | 491.096 | .........................................................................X | 491.096 | | 484.357 | ........................................................................X | 484.357 | | 459.909 | ....................................................................***X | 459.909 | | 447.757 | ...................................................................X | 447.757 | | 423.848 | ...............................................................X | 423.848 | | 422.526 | ...............................................................***X | 422.526 | | 415.046 | ..............................................................X | 415.046 | | 411.120 | .............................................................X | 411.120 | | 394.075 | ..........................................................***X | 394.075 | | 370.338 | .......................................................**..****.......******..*X | 444.902 | | 432.056 | ................................................................***X | 432.056 | | 428.876 | ................................................................*..****X | 440.981 | | 418.269 | ..............................................................X | 418.269 | | 394.063 | ..........................................................****X | 394.063 | | 274.523 | ........................................X | 274.523 | | 240.041 | ...................................*****.......*..***X | 295.922 | | 270.500 | ........................................X | 270.500 | | 265.606 | .......................................X | 265.606 | | 233.790 | ..................................****..........*.............*****.......*..........*****X | 498.990 | | 462.888 | .....................................................................****X | 462.888 | | 388.169 | ..........................................................X | 388.169 | | 333.476 | .................................................****X | 333.476 | | 317.264 | ...............................................X | 317.264 | | 301.062 | ............................................****.........*....................*.............***X | 573.499 | | 563.398 | ....................................................................................X | 563.398 | | 546.865 | ..................................................................................X | 546.865 | | 522.179 | ..............................................................................****..X | 537.280 | | 516.740 | .............................................................................X | 516.740 | | 505.739 | ...........................................................................***X | 505.739 | | 439.635 | .................................................................X | 439.635 | | 390.717 | ..........................................................*****........*....****....*............*.......****....*..X | 657.039 | | 637.388 | ................................................................................................X | 637.388 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:27:40,188 | test | WARNING | Thread-69902 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:w2gQN-51-219000:f5InZUfeYOAjZjr9Tp-51-347000 create failed keys from task: [u'test_collections-568'] 2021-05-17 07:27:40,194 | test | WARNING | Thread-69919 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HyNGMRr3ZuFhkDyMI%3-42-471000:jFwH-42-473000 delete failed keys from task: [u'test_collections-4'] 2021-05-17 07:27:40,220 | test | WARNING | Thread-70030 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:d-JNe0%b_Xp4-52-541000 create failed keys from task: [u'test_collections-1919', u'test_collections-1889', u'test_collections-1672'] 2021-05-17 07:27:40,230 | test | WARNING | Thread-70071 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:G9_dHSfJq8MN%XS-49-719000:T-49-852000 create failed keys from task: [u'test_collections-1919', u'test_collections-1889', u'test_collections-1672'] 2021-05-17 07:27:40,230 | test | WARNING | Thread-70072 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:G9_dHSfJq8MN%XS-49-719000:vjwsBdkHXDy-49-853000 create failed keys from task: [u'test_collections-1919', u'test_collections-1889', u'test_collections-1672'] 2021-05-17 07:27:40,230 | test | WARNING | Thread-70074 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:VJ8HTV%2eeR6724LLF6-49-207000:7mHULQQF0TpwanqG3u7-49-428000 create failed keys from task: [u'test_collections-1919', u'test_collections-1889', u'test_collections-1672'] 2021-05-17 07:27:40,230 | test | WARNING | Thread-70075 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:J-50-71000:f%%tq4uxAv4P-50-179000 create failed keys from task: [u'test_collections-1919', u'test_collections-1889', u'test_collections-1672'] 2021-05-17 07:27:40,243 | test | WARNING | Thread-70116 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:G2nunngeY9_a%BmpMOJT-42-476000 delete failed keys from task: [u'test_collections-522', u'test_collections-567', u'test_collections-886', u'test_collections-762', u'test_collections-562', u'test_collections-681', u'test_collections-614', u'test_collections-538', u'test_collections-653', u'test_collections-829', u'test_collections-708', u'test_collections-701', u'test_collections-500', u'test_collections-501', u'test_collections-862', u'test_collections-541', u'test_collections-660', u'test_collections-715', u'test_collections-716', u'test_collections-714', u'test_collections-513', u'test_collections-874', u'test_collections-512', u'test_collections-752'] 2021-05-17 07:27:40,247 | test | WARNING | Thread-70129 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:bwY-42-476000 update failed keys from task: [u'test_collections-1391', u'test_collections-1103', u'test_collections-1225', u'test_collections-1028', u'test_collections-1226', u'test_collections-1305', u'test_collections-1426', u'test_collections-1229', u'test_collections-1109', u'test_collections-1428', u'test_collections-1429', u'test_collections-1261', u'test_collections-1065', u'test_collections-1264', u'test_collections-1221', u'test_collections-1342', u'test_collections-1136', u'test_collections-1019', u'test_collections-1415', u'test_collections-1295', u'test_collections-1252', u'test_collections-1099', u'test_collections-1134', u'test_collections-1093', u'test_collections-1005', u'test_collections-980', u'test_collections-1129', u'test_collections-1040', u'test_collections-1002', u'test_collections-1288', u'test_collections-1366', u'test_collections-1433', u'test_collections-1318', u'test_collections-1074', u'test_collections-1033', u'test_collections-1198'] 2021-05-17 07:27:40,230 | test | WARNING | Thread-70076 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:J-50-71000:bhRjfqtBD%tE-50-179000 create failed keys from task: [u'test_collections-96'] 2021-05-17 07:27:40,257 | test | WARNING | Thread-70150 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:UD6owGuXFA-21-18000:I-p-21-129000 delete failed keys from task: [u'test_collections-907', u'test_collections-948', u'test_collections-828', u'test_collections-603', u'test_collections-823', u'test_collections-942', u'test_collections-745', u'test_collections-624', u'test_collections-866', u'test_collections-885', u'test_collections-743', u'test_collections-980', u'test_collections-915', u'test_collections-857', u'test_collections-810', u'test_collections-975', u'test_collections-897', u'test_collections-851', u'test_collections-793', u'test_collections-750'] 2021-05-17 07:27:40,263 | test | WARNING | Thread-70163 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pRBhm9zlg_-42-481000:uNAIV_gP%Qw0-42-483000 update failed keys from task: [u'test_collections-1071', u'test_collections-1269', u'test_collections-1225', u'test_collections-1305', u'test_collections-1229', u'test_collections-1307', u'test_collections-1308', u'test_collections-1261', u'test_collections-1340', u'test_collections-1264', u'test_collections-1380', u'test_collections-1059', u'test_collections-1415', u'test_collections-1417', u'test_collections-1173', u'test_collections-1295', u'test_collections-1297', u'test_collections-1298', u'test_collections-1050', u'test_collections-989', u'test_collections-1085', u'test_collections-1241', u'test_collections-1003', u'test_collections-1366', u'test_collections-1115', u'test_collections-1435', u'test_collections-1031', u'test_collections-1076'] 2021-05-17 07:27:40,246 | test | WARNING | Thread-70117 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ub4p8k-42-474000 delete failed keys from task: [u'test_collections-728', u'test_collections-649', u'test_collections-803', u'test_collections-966', u'test_collections-769', u'test_collections-569', u'test_collections-881', u'test_collections-815', u'test_collections-615', u'test_collections-978', u'test_collections-652', u'test_collections-892', u'test_collections-893', u'test_collections-629', u'test_collections-504', u'test_collections-623', u'test_collections-744', u'test_collections-865', u'test_collections-545', u'test_collections-542', u'test_collections-784', u'test_collections-584', u'test_collections-860', u'test_collections-759', u'test_collections-518', u'test_collections-835', u'test_collections-711', u'test_collections-677', u'test_collections-556', u'test_collections-631', u'test_collections-598', u'test_collections-595', u'test_collections-552', u'test_collections-592'] 2021-05-17 07:27:40,250 | test | WARNING | Thread-70139 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:oGdUcQaM-42-475000 delete failed keys from task: [u'test_collections-929', u'test_collections-936', u'test_collections-925', u'test_collections-933', u'test_collections-967', u'test_collections-976', u'test_collections-960'] 2021-05-17 07:27:40,295 | test | WARNING | Thread-70111 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:mVQ1B8cdp7Ds--19-279000:e2Xlf5mzWTIlP-19-404000 update failed keys from task: [u'test_collections-1802'] 2021-05-17 07:27:40,293 | test | WARNING | Thread-70166 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pRBhm9zlg_-42-481000:u%D_OqSAXI-42-484000 update failed keys from task: [u'test_collections-1382', u'test_collections-1386', u'test_collections-1388'] 2021-05-17 07:27:40,263 | test | WARNING | Thread-70157 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:T-51-504000:E0QGUnKYDXwoR5iGJ-51-708000 create failed keys from task: [u'test_collections-939'] 2021-05-17 07:27:40,292 | test | WARNING | Thread-70154 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:4yNQhJ9G%3T_mgMzCoMP-51-876000:qnUnPKQu9KS6FRLVU-51-987000 create failed keys from task: [u'test_collections-858', u'test_collections-740'] 2021-05-17 07:27:40,303 | test | WARNING | Thread-70112 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:UsY-19-603000:qr-19-716000 update failed keys from task: [u'test_collections-1802'] 2021-05-17 07:27:40,309 | test | WARNING | Thread-70111 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:mVQ1B8cdp7Ds--19-279000:e2Xlf5mzWTIlP-19-404000 delete failed keys from task: [u'test_collections-828', u'test_collections-836', u'test_collections-936', u'test_collections-826', u'test_collections-925', u'test_collections-857', u'test_collections-950'] 2021-05-17 07:27:40,326 | test | WARNING | Thread-70112 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:UsY-19-603000:qr-19-716000 delete failed keys from task: [u'test_collections-28', u'test_collections-169', u'test_collections-120', u'test_collections-165', u'test_collections-130'] 2021-05-17 07:27:40,328 | test | WARNING | Thread-70170 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pRBhm9zlg_-42-481000:doeEoQtuCpmJx2-42-483000 update failed keys from task: [u'test_collections-1090', u'test_collections-1170', u'test_collections-1108', u'test_collections-1041', u'test_collections-1100', u'test_collections-1145', u'test_collections-1081', u'test_collections-1030', u'test_collections-1098', u'test_collections-1111', u'test_collections-1057'] 2021-05-17 07:27:40,348 | test | WARNING | Thread-70188 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pRBhm9zlg_-42-481000:usGy5YX7o8-42-482000 update failed keys from task: [u'test_collections-1228', u'test_collections-1263', u'test_collections-1341', u'test_collections-1378', u'test_collections-1137', u'test_collections-1016', u'test_collections-1172', u'test_collections-1132', u'test_collections-1013', u'test_collections-1014', u'test_collections-1377', u'test_collections-1048', u'test_collections-982', u'test_collections-1049', u'test_collections-1369', u'test_collections-1327', u'test_collections-1240', u'test_collections-1284', u'test_collections-1362', u'test_collections-1165', u'test_collections-1245', u'test_collections-1314', u'test_collections-1271', u'test_collections-1152', u'test_collections-1354', u'test_collections-1234'] 2021-05-17 07:27:40,354 | test | WARNING | Thread-70221 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:zF0-42-477000 update failed keys from task: [u'test_collections-1379', u'test_collections-1204', u'test_collections-1326', u'test_collections-1205', u'test_collections-1370', u'test_collections-1382', u'test_collections-1363', u'test_collections-1386', u'test_collections-1388'] 2021-05-17 07:27:40,371 | test | WARNING | Thread-70228 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k%qJV6Btrv-w-42-484000:-L%eaCoIOm4-42-485000 update failed keys from task: [u'test_collections-1411', u'test_collections-1402', u'test_collections-1394', u'test_collections-1395'] 2021-05-17 07:27:40,523 | test | WARNING | Thread-70238 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k%qJV6Btrv-w-42-484000:PDil-42-486000 update failed keys from task: [u'test_collections-1090', u'test_collections-1170', u'test_collections-1128', u'test_collections-1029', u'test_collections-1108', u'test_collections-1084', u'test_collections-1020', u'test_collections-1043', u'test_collections-1100', u'test_collections-1081', u'test_collections-1114', u'test_collections-1159', u'test_collections-1075', u'test_collections-1098', u'test_collections-1055', u'test_collections-1133', u'test_collections-1056', u'test_collections-1111'] 2021-05-17 07:27:40,555 | test | WARNING | Thread-70295 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:1XB2iUVdVou-42-479000 update failed keys from task: [u'test_collections-1379', u'test_collections-1412', u'test_collections-1418', u'test_collections-1395', u'test_collections-1387', u'test_collections-1388'] 2021-05-17 07:27:40,559 | test | WARNING | Thread-70199 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:SB5X1DOr2xL-42-476000 delete failed keys from task: [u'test_collections-606', u'test_collections-603', u'test_collections-885', u'test_collections-539', u'test_collections-979', u'test_collections-935', u'test_collections-857', u'test_collections-810', u'test_collections-734', u'test_collections-610', u'test_collections-731', u'test_collections-773', u'test_collections-971', u'test_collections-851', u'test_collections-908', u'test_collections-828', u'test_collections-823', u'test_collections-624', u'test_collections-866', u'test_collections-621', u'test_collections-664', u'test_collections-713', u'test_collections-954', u'test_collections-951', u'test_collections-710'] 2021-05-17 07:27:40,605 | test | WARNING | Thread-70200 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collections_2 update failed keys from task: [u'test_collections-1204', u'test_collections-1325', u'test_collections-1205', u'test_collections-1360', u'test_collections-1363', u'test_collections-1089', u'test_collections-1080', u'test_collections-1235', u'test_collections-1279', u'test_collections-1237', u'test_collections-1414', u'test_collections-1239', u'test_collections-1119', u'test_collections-1339', u'test_collections-1230', u'test_collections-1110', u'test_collections-1253', u'test_collections-1432'] 2021-05-17 07:27:40,765 | test | WARNING | Thread-70241 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k%qJV6Btrv-w-42-484000:uYqhmbMr-42-484000 update failed keys from task: [u'test_collections-1390', u'test_collections-1423', u'test_collections-1149', u'test_collections-1348', u'test_collections-1304', u'test_collections-1349', u'test_collections-1187', u'test_collections-1341', u'test_collections-1421', u'test_collections-1180', u'test_collections-1137', u'test_collections-1294', u'test_collections-1176', u'test_collections-988', u'test_collections-1367', u'test_collections-1247', u'test_collections-1369', u'test_collections-1008', u'test_collections-1161', u'test_collections-1045', u'test_collections-1082', u'test_collections-994', u'test_collections-1073', u'test_collections-1196', u'test_collections-1199', u'test_collections-1156', u'test_collections-1112', u'test_collections-1354'] 2021-05-17 07:27:40,805 | test | WARNING | Thread-70228 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:k%qJV6Btrv-w-42-484000:-L%eaCoIOm4-42-485000 delete failed keys from task: [u'test_collections-708', u'test_collections-889', u'test_collections-568', u'test_collections-500', u'test_collections-685', u'test_collections-562', u'test_collections-619', u'test_collections-636', u'test_collections-538', u'test_collections-533', u'test_collections-874', u'test_collections-797', u'test_collections-831', u'test_collections-576'] 2021-05-17 07:27:40,898 | test | WARNING | Thread-70295 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:1XB2iUVdVou-42-479000 delete failed keys from task: [u'test_collections-607'] 2021-05-17 07:27:41,897 | test | WARNING | Thread-70200 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collections_2 delete failed keys from task: [u'test_collections-907', u'test_collections-948', u'test_collections-729', u'test_collections-942', u'test_collections-844', u'test_collections-841', u'test_collections-760', u'test_collections-780', u'test_collections-819', u'test_collections-915', u'test_collections-638', u'test_collections-836', u'test_collections-975', u'test_collections-833', u'test_collections-796', u'test_collections-873', u'test_collections-793', u'test_collections-750'] 2021-05-17 07:27:53,174 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 07:28:55,065 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 07:28:55,066 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 07:29:12,732 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 07:29:12,733 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 07:29:31,467 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 07:29:31,467 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 07:30:32,619 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 07:30:32,634 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 Ran 1 test in 3493.028s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 20 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_20 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_21 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_21', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 21, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 07:30:42,599 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #21 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 07:30:43,305 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #21 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 07:30:43,740 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xbe203b type=client open_count=1 channel=[id: 0x664d1717, 0.0.0.0/0.0.0.0:37685] timeout=300.0>) 2021-05-17 07:31:04,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xbe203c type=client open_count=1 channel=[id: 0xc0c634d2, 0.0.0.0/0.0.0.0:54784] timeout=300.0>) 2021-05-17 07:31:06,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xbe203d type=client open_count=1 channel=[id: 0xbed5d0ff, 0.0.0.0/0.0.0.0:60661] timeout=300.0>) 2021-05-17 07:31:08,625 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xbe203e type=client open_count=1 channel=[id: 0x192ebfeb, 0.0.0.0/0.0.0.0:37633] timeout=300.0>) 2021-05-17 07:31:32,974 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xbe203f type=client open_count=1 channel=[id: 0xb27f2674, 0.0.0.0/0.0.0.0:51262] timeout=300.0>) 2021-05-17 07:31:34,993 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xbe2040 type=client open_count=1 channel=[id: 0xe329148d, 0.0.0.0/0.0.0.0:44153] timeout=300.0>) 2021-05-17 07:31:37,009 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xbe2041 type=client open_count=1 channel=[id: 0x75566a03, 0.0.0.0/0.0.0.0:59551] timeout=300.0>) 2021-05-17 07:32:01,155 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xbe2042 type=client open_count=1 channel=[id: 0xcf60b96d, 0.0.0.0/0.0.0.0:37243] timeout=300.0>) 2021-05-17 07:32:03,174 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xbe2043 type=client open_count=1 channel=[id: 0xdb157029, 0.0.0.0/0.0.0.0:52622] timeout=300.0>) 2021-05-17 07:32:05,190 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xbe2044 type=client open_count=1 channel=[id: 0xc8163bde, 0.0.0.0/0.0.0.0:51988] timeout=300.0>) 2021-05-17 07:32:29,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xbe2045 type=client open_count=1 channel=[id: 0x27fdc2a7, 0.0.0.0/0.0.0.0:51905] timeout=300.0>) 2021-05-17 07:32:31,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xbe2046 type=client open_count=1 channel=[id: 0xda855163, 0.0.0.0/0.0.0.0:59678] timeout=300.0>) 2021-05-17 07:32:33,619 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xbe2047 type=client open_count=1 channel=[id: 0x2a0afe3e, 0.0.0.0/0.0.0.0:32992] timeout=300.0>) 2021-05-17 07:32:58,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xbe2048 type=client open_count=1 channel=[id: 0xf0dc234b, 0.0.0.0/0.0.0.0:39069] timeout=300.0>) 2021-05-17 07:33:00,124 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xbe2049 type=client open_count=1 channel=[id: 0xea8dac0c, 0.0.0.0/0.0.0.0:37242] timeout=300.0>) 2021-05-17 07:33:02,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 07:33:19,194 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 07:33:32,914 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 07:33:32,940 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #21 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 07:33:32,980 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #21 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 07:33:46,062 | test | INFO | pool-23-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.311671861198 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 07:33:46,088 | test | INFO | pool-23-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 07:33:56,161 | test | INFO | pool-23-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 07:33:56,187 | test | INFO | pool-23-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-17 07:33:56,437 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 07:34:41,023 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 07:34:41,318 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 728562072 | 69938658 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 702933920 | 99415314 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 1144109336 | 32053599 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 07:34:41,319 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 07:35:27,454 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 07:35:27,471 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 07:51:41,960 | test | INFO | pool-23-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****....................................................................................................*****X | 763.566 | | 701.053 | ..........................................................................................*..***X | 713.289 | | 705.009 | ...........................................................................................X | 705.009 | | 587.505 | ...........................................................................****X | 587.505 | | 583.069 | ...........................................................................*.****X | 595.562 | | 574.379 | ..........................................................................X | 574.379 | | 561.986 | ........................................................................*****...........*****.X | 656.637 | | 655.540 | ....................................................................................****X | 655.540 | | 622.900 | ................................................................................X | 622.900 | | 597.944 | .............................................................................****X | 597.944 | | 550.954 | .......................................................................X | 550.954 | | 543.057 | ......................................................................****X | 543.057 | | 463.030 | ...........................................................X | 463.030 | | 444.737 | .........................................................****...*...*****X | 496.244 | | 491.002 | ...............................................................****X | 491.002 | | 466.914 | ............................................................******........******....*..****X | 577.701 | | 481.904 | ..............................................................******.****X | 489.123 | | 481.903 | ..............................................................******....*****X | 515.811 | | 449.591 | .........................................................X | 449.591 | | 445.726 | .........................................................*****.....*****...*........****X | 568.788 | | 439.340 | ........................................................X | 439.340 | | 396.737 | ..................................................*****..........*......****X | 518.722 | | 494.818 | ...............................................................****X | 494.818 | | 456.125 | ..........................................................X | 456.125 | | 337.644 | ...........................................*****...*..........................****X | 557.418 | | 543.429 | ......................................................................*.****X | 549.935 | | 513.688 | ..................................................................*****X | 513.688 | | 460.135 | ...........................................................*****...*......****X | 529.237 | | 461.205 | ...........................................................*****...*..........****X | 557.490 | | 542.849 | ......................................................................****X | 542.849 | | 529.880 | ....................................................................X | 529.880 | | 510.536 | .................................................................****X | 510.536 | | 499.516 | ................................................................X | 499.516 | | 457.426 | ..........................................................*****.......*****X | 507.631 | | 482.372 | ..............................................................X | 482.372 | | 411.567 | ....................................................****.*.......****X | 471.340 | | 464.740 | ...........................................................*.......*****..X | 529.339 | | 475.885 | .............................................................*****....******..X | 524.090 | | 516.420 | ..................................................................***X | 516.420 | | 497.558 | ................................................................X | 497.558 | | 442.927 | .........................................................*****....*****X | 474.826 | | 421.451 | ......................................................*.****...*.........*****.X | 532.372 | | 480.082 | .............................................................****X | 480.082 | | 429.506 | .......................................................X | 429.506 | | 410.894 | ....................................................*****............*******.......****.X | 558.917 | | 513.159 | ..................................................................****X | 513.159 | | 491.859 | ...............................................................X | 491.859 | | 442.038 | ........................................................*****.....X | 477.585 | | 453.450 | ..........................................................****X | 453.450 | | 428.149 | .......................................................*.......*****.............*****X | 583.026 | | 531.981 | ....................................................................*..***X | 548.181 | | 515.956 | ..................................................................X | 515.956 | | 446.162 | .........................................................****X | 446.162 | | 438.492 | ........................................................*....*****.......******......****X | 566.331 | | 466.657 | ............................................................X | 466.657 | | 440.834 | ........................................................*****.......*.....***X | 533.406 | | 526.016 | ...................................................................X | 526.016 | | 469.060 | ............................................................*****.....******....****X | 537.048 | | 463.392 | ...........................................................X | 463.392 | | 422.503 | ......................................................X | 422.503 | | 410.705 | ....................................................****...........*..****X | 504.964 | | 485.721 | ..............................................................X | 485.721 | | 468.057 | ............................................................****...*..*****...****X | 527.184 | | 494.507 | ...............................................................X | 494.507 | | 479.307 | .............................................................X | 479.307 | | 476.682 | .............................................................****.......*.....****X | 572.201 | | 454.491 | ..........................................................****X | 454.491 | | 451.800 | ..........................................................*...*****.....X | 514.716 | | 513.861 | ..................................................................*****....*****X | 546.904 | | 505.758 | .................................................................*****X | 505.758 | | 467.794 | ............................................................******....................****X | 620.529 | | 548.736 | ......................................................................X | 548.736 | | 483.686 | ..............................................................****X | 483.686 | | 458.993 | ...........................................................X | 458.993 | | 457.726 | ..........................................................*****...........*****X | 535.192 | | 480.293 | .............................................................*****.......*....****X | 561.601 | | 557.289 | .......................................................................*...****X | 573.996 | | 515.671 | ..................................................................*****..*.****X | 536.697 | | 486.817 | ..............................................................****X | 486.817 | | 483.399 | ..............................................................X | 483.399 | | 459.793 | ...........................................................****X | 459.793 | | 451.535 | ..........................................................*..*****........****X | 534.326 | | 508.547 | .................................................................******.............*...****.X | 640.259 | | 621.773 | ................................................................................*****.X | 628.385 | | 623.197 | ................................................................................****X | 623.197 | | 564.155 | ........................................................................**X | 564.155 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 07:51:42,611 | test | INFO | pool-23-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****....*...........****X | 4.499 | | 1.199 | ...X | 1.199 | | 0.000 | ***************************************...*..*****..........******...*..***X | 5.752 | | 4.299 | ..............X | 4.299 | | 1.500 | ....*****......*...........****X | 6.299 | | 5.699 | ...................X | 5.699 | | 2.917 | .........****X | 2.917 | | 2.217 | ......X | 2.217 | | 1.099 | ..*****...****X | 1.900 | | 1.900 | .....*....*****X | 2.899 | | 0.100 | ****X | 0.100 | | 0.000 | ************.......*...****X | 3.003 | | 0.000 | *****.....*.....................****X | 7.505 | | 6.905 | .......................X | 6.905 | | 4.504 | ..............*****........*...****X | 7.505 | | 4.502 | ..............*****X | 4.502 | | 3.003 | .........X | 3.003 | | 1.102 | ..****....*....*****....................*****X | 9.005 | | 2.400 | .......X | 2.400 | | 0.000 | *********************.....************.......*.........***X | 6.003 | | 5.102 | ................X | 5.102 | | 1.501 | ....****X | 1.501 | | 1.201 | ...X | 1.201 | | 0.000 | *****...............*****X | 4.502 | | 0.000 | ***********..*...****X | 1.501 | | 1.501 | ....*****....*.****...*........*****....*.....****X | 8.701 | | 3.302 | ..........*****X | 3.302 | | 3.002 | .........X | 3.002 | | 3.001 | .........***X | 3.001 | | 2.701 | ........X | 2.701 | | 1.500 | ....*****....*.................****X | 7.502 | | 0.000 | ***********............*...................................................****X | 18.001 | | 10.304 | ...................................X | 10.304 | | 1.500 | ....*****....................*..........................****X | 14.503 | | 11.703 | ........................................X | 11.703 | | 0.500 | *****.........*****X | 3.000 | | 1.501 | ....*****.....*.....****X | 4.500 | | 4.201 | .............X | 4.201 | | 3.001 | .........****X | 3.001 | | 3.001 | .........****X | 3.001 | | 3.001 | .........X | 3.001 | | 3.000 | .........****X | 3.000 | | 2.300 | .......X | 2.300 | | 1.501 | ....****........*..****X | 4.499 | | 2.000 | ......X | 2.000 | | 1.201 | ...****.....*.............****X | 6.292 | | 6.008 | ....................****X | 6.008 | | 3.908 | ............X | 3.908 | | 3.100 | .........*****...........*..........***X | 8.902 | | 6.402 | .....................X | 6.402 | | 0.000 | ***********.........****X | 2.700 | | 2.300 | .......X | 2.300 | | 2.200 | ......X | 2.200 | | 1.800 | .....***X | 1.800 | | 1.800 | .....X | 1.800 | | 1.500 | ....*****X | 1.500 | | 1.500 | ....****X | 1.500 | | 0.000 | ***********..........................*...............................................*...........................***X | 28.406 | | 16.702 | .........................................................X | 16.702 | | 3.100 | .........****X | 3.100 | | 1.600 | ....X | 1.600 | | 0.000 | *****..*...*****..*...***X | 3.000 | | 2.700 | ........X | 2.700 | | 1.900 | .....*****...*..........****X | 5.600 | | 1.500 | ....******.....*****.......................*.........****X | 12.000 | | 4.500 | ..............****X | 4.500 | | 4.500 | ..............X | 4.500 | | 4.498 | ..............*****.....................................*****X | 15.008 | | 0.000 | *************************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 07:51:43,365 | test | INFO | pool-23-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****......*..............................................................................................****X | 65.985 | | 17.591 | .........................X | 17.591 | | 0.000 | ***********************....****X | 3.000 | | 0.800 | X | 0.800 | | 0.000 | ***********.*.*****....*****..............*.....****X | 16.643 | | 8.798 | ............X | 8.798 | | 6.001 | ........****.*.....****X | 10.498 | | 7.799 | ..........X | 7.799 | | 1.159 | *****.*..*****..*...*****.*****X | 7.098 | | 5.099 | ......X | 5.099 | | 1.899 | .*****..*****.*....****X | 6.006 | | 5.207 | ......X | 5.207 | | 3.003 | ...****X | 3.003 | | 2.403 | ..X | 2.403 | | 0.000 | *****...........*****X | 7.505 | | 0.000 | *****.*.....****X | 4.503 | | 1.501 | .*****X | 1.501 | | 0.000 | *****..*.........****X | 7.503 | | 4.199 | .....X | 4.199 | | 3.002 | ...****X | 3.002 | | 0.000 | ************....****X | 3.002 | | 0.800 | X | 0.800 | | 0.000 | *****.............*****X | 9.005 | | 8.905 | ............****X | 8.905 | | 8.004 | ...........X | 8.004 | | 7.603 | ..........****X | 7.603 | | 4.004 | .....X | 4.004 | | 0.000 | *********************************************....*****.*.****X | 4.500 | | 3.002 | ...*****X | 3.002 | | 1.500 | .******.........*****X | 7.502 | | 4.501 | .....****X | 4.501 | | 2.401 | ..X | 2.401 | | 0.000 | *****..******....*****X | 4.500 | | 0.000 | ******....****X | 3.000 | | 2.601 | ..X | 2.601 | | 1.500 | .*****....******..............****X | 13.505 | | 11.505 | ................X | 11.505 | | 8.003 | ...........X | 8.003 | | 6.001 | ........***X | 6.001 | | 4.801 | ......X | 4.801 | | 0.000 | *****..*****X | 1.500 | | 0.000 | *****.*.****X | 1.501 | | 0.400 | X | 0.400 | | 0.000 | *****.*.***X | 1.501 | | 0.801 | X | 0.801 | | 0.000 | *****....****X | 3.004 | | 2.204 | ..X | 2.204 | | 0.000 | ***********.*.****X | 1.500 | | 0.000 | ***********.*........****..*..****X | 8.600 | | 6.501 | ........X | 6.501 | | 4.900 | ......***X | 4.900 | | 4.500 | .....X | 4.500 | | 3.000 | ...****X | 3.000 | | 1.600 | .X | 1.600 | | 0.000 | *****.*.****X | 1.500 | | 1.200 | X | 1.200 | | 0.000 | ***********************.*****.*****X | 1.800 | | 1.500 | .*****X | 1.500 | | 0.000 | *****......*...****X | 6.000 | | 0.000 | ***********......*...***X | 6.003 | | 1.602 | .X | 1.602 | | 0.000 | *************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 07:51:43,619 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 73.0207673589 | 6437847040 | 4347191296 | 2398937088 / 7511588864 | 101707 / 204781 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 72.735 | 6437847040 | 4267139072 | 2455105536 / 7511588864 | 99919 / 202643 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 79.0366666667 | 6437847040 | 4246978560 | 2452258816 / 7511588864 | 100285 / 202134 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 60.3116666667 | 6437847040 | 3836055552 | 3053629440 / 9530753024 | 101546 / 199195 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 69.765 | 6437847040 | 4255309824 | 2453200896 / 7511588864 | 101060 / 200236 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 07:51:56,273 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 07:53:22,539 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164970104 | 240500345 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162158016 | 274048949 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 444994936 | 451271050 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 07:53:22,539 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #21 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 07:53:32,631 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 07:53:32,632 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 07:53:32,634 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 07:54:15,023 | test | INFO | pool-23-thread-1 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 42.2009999752 seconds to complete 2021-05-17 07:54:15,028 | test | INFO | pool-23-thread-1 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 07:56:20,085 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0520000457764 seconds 2021-05-17 07:56:20,085 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 07:56:20,085 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 07:56:26,683 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 08:07:04,305 | test | INFO | pool-23-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****................*.................................................*****X | 718.323 | | 551.786 | .................................................*****X | 551.786 | | 469.773 | .........................................******..........................................................*****X | 1102.904 | | 344.622 | ..............................******...............*****X | 518.142 | | 359.963 | ...............................****X | 359.963 | | 348.174 | ..............................X | 348.174 | | 341.449 | .............................****X | 341.449 | | 337.041 | .............................X | 337.041 | | 293.963 | .........................****X | 293.963 | | 282.344 | ........................X | 282.344 | | 263.787 | ......................*****.....*...........****X | 440.788 | | 407.997 | ...................................X | 407.997 | | 393.875 | ..................................****X | 393.875 | | 372.448 | ................................*****X | 372.448 | | 359.120 | ...............................******..******.....................*............****X | 741.891 | | 653.500 | ..........................................................X | 653.500 | | 489.415 | ...........................................****X | 489.415 | | 456.602 | ........................................******.X | 471.655 | | 461.343 | ........................................****X | 461.343 | | 384.056 | .................................X | 384.056 | | 369.026 | ................................*****...X | 399.480 | | 397.751 | ...................................****...*....*****.X | 488.925 | | 475.896 | ..........................................*****....*.****X | 535.424 | | 466.820 | .........................................X | 466.820 | | 430.264 | ......................................****X | 430.264 | | 406.674 | ...................................*..*****.....*****X | 482.357 | | 473.451 | .........................................*****X | 473.451 | | 389.772 | ..................................*****X | 389.772 | | 298.400 | ..........................**X | 298.400 | | 298.400 | ..........................***.......*****X | 381.708 | | 322.213 | ............................X | 322.213 | | 319.638 | ...........................*****...*....****X | 387.614 | | 359.845 | ...............................******.X | 370.572 | | 370.572 | ................................**X | 370.572 | | 370.572 | ................................**.*..*****.*.....*****........*......*X | 620.663 | | 620.663 | .......................................................*X | 620.663 | | 620.663 | .......................................................X | 620.663 | | 545.500 | ................................................X | 545.500 | | 537.720 | ...............................................****..X | 552.648 | | 473.920 | .........................................*****...........*..........****X | 695.585 | | 586.585 | ....................................................X | 586.585 | | 549.215 | ................................................****X | 549.215 | | 545.176 | ................................................X | 545.176 | | 538.416 | ...............................................*****.******........*****X | 631.356 | | 550.470 | ................................................X | 550.470 | | 514.886 | .............................................*****......*****X | 580.287 | | 571.940 | ..................................................****X | 571.940 | | 566.639 | ..................................................*..*****X | 591.653 | | 490.781 | ...........................................******......***********X | 555.957 | | 531.319 | ...............................................******.....*****X | 592.078 | | 555.050 | .................................................******..*****X | 582.151 | | 553.643 | .................................................******.....*....****X | 657.377 | | 517.487 | .............................................******.....******....*****X | 609.939 | | 602.130 | .....................................................******.................***X | 787.180 | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 08:07:05,638 | test | INFO | pool-23-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****..*........*****.......................................*.....................*****......X | 160.801 | | 158.092 | ..........................................................................****X | 158.092 | | 157.399 | ..........................................................................******..******........****X | 179.215 | | 175.406 | ..................................................................................X | 175.406 | | 175.382 | ..................................................................................*****.*..****X | 180.328 | | 167.617 | ...............................................................................X | 167.617 | | 151.381 | .......................................................................*****..*......*.......***X | 183.891 | | 173.590 | .................................................................................X | 173.590 | | 126.839 | ...........................................................******.................*****X | 162.482 | | 152.181 | .......................................................................*****........*...................**.X | 209.513 | | 209.513 | ..................................................................................................*X | 209.513 | | 193.319 | ...........................................................................................X | 193.319 | | 153.599 | ........................................................................X | 153.599 | | 117.618 | .......................................................***X | 117.618 | | 105.109 | .................................................X | 105.109 | | 90.541 | ..........................................*****.*...X | 98.843 | | 96.204 | ............................................****......**.****...**.****.*..****X | 124.209 | | 113.299 | .....................................................X | 113.299 | | 76.391 | ...................................*****.*...****X | 84.171 | | 66.527 | ..............................X | 66.527 | | 60.059 | ...........................****X | 60.059 | | 14.836 | ......X | 14.836 | | 1.197 | *****.*****X | 3.492 | | 2.593 | X | 2.593 | | 1.697 | *****.*****X | 3.494 | | 2.494 | X | 2.494 | | 1.697 | **********X | 1.896 | | 0.899 | ****X | 0.899 | | 0.799 | X | 0.799 | | 0.699 | ****X | 0.699 | | 0.599 | X | 0.599 | | 0.499 | ************.**********X | 2.697 | | 2.497 | X | 2.497 | | 1.898 | ******.******.******X | 5.993 | | 5.893 | .***X | 5.893 | | 4.493 | .X | 4.493 | | 0.899 | ******.*****X | 2.997 | | 2.798 | ****X | 2.798 | | 2.698 | ************.*****X | 4.797 | | 2.198 | ************X | 2.598 | | 2.598 | ****X | 2.598 | | 1.300 | ******..*****X | 4.197 | | 4.097 | X | 4.097 | | 3.898 | *****X | 3.898 | | 3.298 | ***********.*****X | 6.196 | | 5.796 | .X | 5.796 | | 4.298 | .*****X | 4.298 | | 3.599 | *****X | 3.599 | | 3.599 | ******.*****X | 5.097 | | 1.499 | ******..*****X | 4.597 | | 4.198 | .***********X | 6.097 | | 4.799 | .****X | 4.799 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 08:07:05,858 | test | INFO | pool-23-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **********.******.....................*.......****X | 55.799 | | 52.590 | ..........................*....X | 58.400 | | 58.400 | ..............................****...*...*****...X | 76.105 | | 71.706 | .....................................****X | 71.706 | | 70.092 | ....................................*..*****......*..*****..X | 93.907 | | 86.889 | .............................................****X | 86.889 | | 84.083 | ...........................................X | 84.083 | | 78.897 | ........................................****X | 78.897 | | 73.765 | ......................................******.....*****X | 84.441 | | 70.692 | ....................................*****.X | 72.700 | | 68.804 | ...................................****X | 68.804 | | 65.207 | .................................*...X | 69.601 | | 63.509 | ................................****..*..****X | 70.282 | | 27.881 | .............X | 27.881 | | 7.701 | ...****X | 7.701 | | 1.900 | X | 1.900 | | 0.200 | *********************X | 1.300 | | 1.300 | X | 1.300 | | 1.000 | X | 1.000 | | 0.798 | ****.....*................******............................*****X | 92.693 | | 86.319 | ............................................*****X | 86.319 | | 81.515 | ..........................................*...*****............*..****........*...............****X | 156.452 | | 154.652 | .................................................................................X | 154.652 | | 153.726 | ................................................................................****X | 153.726 | | 148.627 | ..............................................................................*.........X | 166.397 | | 166.397 | .......................................................................................***X | 166.397 | | 155.586 | .................................................................................X | 155.586 | | 153.677 | ................................................................................*****..............*.....****X | 187.962 | | 172.272 | ..........................................................................................*****X | 172.272 | | 162.280 | .....................................................................................*****X | 162.280 | | 160.484 | ....................................................................................X | 160.484 | | 157.512 | ..................................................................................*****..............*****X | 183.585 | | 111.822 | ..........................................................*****X | 111.822 | | 1.598 | *****X | 1.598 | | 1.598 | X | 1.598 | | 1.199 | ******.*****.*****X | 4.695 | | 0.699 | *******.*****.*****X | 4.795 | | 2.499 | *****X | 2.499 | | 1.899 | X | 1.899 | | 0.899 | *****************..*****X | 4.598 | | 3.298 | *****X | 3.298 | | 2.598 | X | 2.598 | | 1.999 | *****..*****X | 6.897 | | 1.199 | ******.*******.****X | 4.198 | | 1.000 | ************..******X | 4.397 | | 2.899 | **********X | 3.398 | | 2.598 | X | 2.598 | | 1.599 | *****.**.***X | 4.598 | | 2.998 | *X | 2.998 | | 1.700 | ***X | 1.700 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 08:07:06,125 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 08:07:06,174 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 08:07:16,496 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 08:07:16,654 | test | INFO | pool-23-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.313328111198 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 39.087969599 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 39.6177280694 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 44.819080318 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 33.7005433152 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 08:07:16,704 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:07:21,796 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:07:24,269 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 08:07:26,857 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:07:31,921 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:07:36,994 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:07:42,076 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 1.0 2021-05-17 08:07:47,183 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 5.11 2021-05-17 08:07:52,262 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 8.93 2021-05-17 08:07:54,915 | test | INFO | pool-23-thread-5 | [table_view:display:72] Keys failed in bucket1:bF-24-444000:S-24-582000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1121 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:07:57,335 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 12.66 2021-05-17 08:08:02,426 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 16.34 2021-05-17 08:08:04,960 | test | INFO | pool-23-thread-5 | [table_view:display:72] Keys failed in bucket1:bF-24-444000:S-24-582000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:07,542 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 19.1 2021-05-17 08:08:12,638 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 23.52 2021-05-17 08:08:17,721 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 26.85 2021-05-17 08:08:22,799 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 30.49 2021-05-17 08:08:27,900 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.08 2021-05-17 08:08:32,994 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 34.91 2021-05-17 08:08:38,101 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 38.43 2021-05-17 08:08:43,223 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 41.83 2021-05-17 08:08:47,605 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in bucket2:Hlpnq-20-563000:JP2k0NDZ7fdE900-20-681000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,607 | test | INFO | pool-23-thread-20 | [table_view:display:72] Keys failed in bucket2:W-20-282000:VgCXaNnuZTXP-20-425000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,607 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in bucket2:Hlpnq-20-563000:JP2k0NDZ7fdE900-20-681000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,607 | test | INFO | pool-23-thread-9 | [table_view:display:72] Keys failed in bucket2:W-20-282000:CJ%nyboOP7Ejd-20-425000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,607 | test | INFO | pool-23-thread-6 | [table_view:display:72] Keys failed in bucket2:Hlpnq-20-563000:4g9wde2a-24-85000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,608 | test | INFO | pool-23-thread-20 | [table_view:display:72] Keys failed in bucket2:W-20-282000:VgCXaNnuZTXP-20-425000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,608 | test | INFO | pool-23-thread-9 | [table_view:display:72] Keys failed in bucket2:W-20-282000:CJ%nyboOP7Ejd-20-425000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:47,608 | test | INFO | pool-23-thread-6 | [table_view:display:72] Keys failed in bucket2:Hlpnq-20-563000:4g9wde2a-24-85000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:48,345 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 45.22 2021-05-17 08:08:52,576 | test | INFO | pool-23-thread-29 | [table_view:display:72] Keys failed in bucket2:OE8-i7V1-20-871000:N-20-990000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:52,578 | test | INFO | pool-23-thread-29 | [table_view:display:72] Keys failed in bucket2:OE8-i7V1-20-871000:N-20-990000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:08:53,450 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 48.76 2021-05-17 08:08:58,546 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 51.0 2021-05-17 08:09:03,635 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 54.24 2021-05-17 08:09:08,720 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 57.83 2021-05-17 08:09:13,831 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 60.27 2021-05-17 08:09:18,911 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 62.93 2021-05-17 08:09:23,982 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 66.1 2021-05-17 08:09:29,068 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 08:09:34,154 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 68.84 2021-05-17 08:09:39,267 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 71.07 2021-05-17 08:09:40,644 | test | INFO | pool-23-thread-24 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1115 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1085 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:41,173 | test | INFO | pool-23-thread-14 | [table_view:display:72] Keys failed in default:_default:zngj-56-534000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1115 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1085 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:41,724 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:_default:F9k7JEZ1OQXr82go-56-534000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:44,394 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 72.88 2021-05-17 08:09:45,831 | test | INFO | pool-23-thread-7 | [table_view:display:72] Keys failed in default:_default:DUdNE3-H4s-56-533000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:46,421 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:nnNMXk_zN2z0bW29ib-56-544000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:46,421 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:scope1:LWCmMGG%khkO-56-537000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:49,516 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 75.19 2021-05-17 08:09:50,144 | test | INFO | pool-23-thread-2 | [table_view:display:72] Keys failed in default:1dnqgOucFPG7KRD9S-22-822000:QLIfDTjdUOLJ15JV-22-936000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:50,145 | test | INFO | pool-23-thread-2 | [table_view:display:72] Keys failed in default:1dnqgOucFPG7KRD9S-22-822000:QLIfDTjdUOLJ15JV-22-936000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1957 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:50,901 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:N9qSlfFZuby%-56-545000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-812 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-830 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-785 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:51,427 | test | INFO | pool-23-thread-16 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:N9qSlfFZuby%-56-545000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:51,773 | test | INFO | pool-23-thread-24 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1348 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:53,506 | test | INFO | pool-23-thread-14 | [table_view:display:72] Keys failed in default:_default:zngj-56-534000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1348 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:54,604 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 77.51 2021-05-17 08:09:55,483 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:_default:F9k7JEZ1OQXr82go-56-534000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:56,052 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:nnNMXk_zN2z0bW29ib-56-544000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:57,631 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:scope1:LWCmMGG%khkO-56-537000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:09:59,709 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 80.01 2021-05-17 08:10:00,328 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:N9qSlfFZuby%-56-545000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-911 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:01,503 | test | INFO | pool-23-thread-3 | [table_view:display:72] Keys failed in default:eeZ-23-758000:VSuGB-23-863000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:01,503 | test | INFO | pool-23-thread-3 | [table_view:display:72] Keys failed in default:eeZ-23-758000:VSuGB-23-863000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1124 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:02,530 | test | INFO | pool-23-thread-24 | [table_view:display:72] Keys failed in default:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:02,614 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:eeZ-23-758000:ADtP-23-863000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2313 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2283 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:02,615 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:eeZ-23-758000:ADtP-23-863000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2313 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2283 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:02,684 | test | INFO | pool-23-thread-7 | [table_view:display:72] Keys failed in default:_default:DUdNE3-H4s-56-533000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:03,150 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:_default:F9k7JEZ1OQXr82go-56-534000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1417 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:04,010 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:Dk8ik4-56-543000:nnNMXk_zN2z0bW29ib-56-544000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1430 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 08:10:04,812 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 81.82 2021-05-17 08:10:09,900 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 83.59 2021-05-17 08:10:15,003 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 85.36 2021-05-17 08:10:20,098 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 87.28 2021-05-17 08:10:25,177 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 89.11 2021-05-17 08:10:30,250 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 90.61 2021-05-17 08:10:35,332 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 92.28 2021-05-17 08:10:40,427 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 93.84 2021-05-17 08:10:45,503 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 95.56 2021-05-17 08:10:50,665 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 97.38 2021-05-17 08:10:55,755 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: running, progress: 98.77 2021-05-17 08:11:05,861 | test | INFO | pool-23-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 08:11:05,910 | test | INFO | pool-23-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 229.255000114 sec 2021-05-17 08:12:47,918 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 08:12:48,497 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:12:48,500 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:12:48,500 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:13:13,743 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:13:13,744 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:13:13,747 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 26 docs from 1000 to 1960 2021-05-17 08:14:31,917 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:14:31,918 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 08:14:31,921 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 08:14:31,921 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 8 docs from 980 to 1440 2021-05-17 08:14:31,921 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-17 08:14:31,923 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 15 docs from 980 to 1440 2021-05-17 08:14:31,923 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 21 docs from 980 to 1440 2021-05-17 08:14:31,924 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 16 docs from 980 to 1440 2021-05-17 08:14:31,924 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 13 docs from 980 to 1440 2021-05-17 08:14:31,924 | test | ERROR | pool-23-thread-8 | [task:call:3809] Failed to load 11 docs from 980 to 1440 2021-05-17 08:21:19,265 | test | INFO | pool-23-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******....................................................................................................****X | 1202.491 | | 301.573 | ........................****X | 301.573 | | 208.498 | ................****X | 208.498 | | 190.728 | ..............****....................*.......***X | 506.176 | | 241.270 | ...................****X | 241.270 | | 233.376 | ..................****..............*.............................................**X | 940.593 | | 940.593 | .............................................................................X | 940.593 | | 734.078 | ............................................................X | 734.078 | | 229.807 | ..................***X | 229.807 | | 173.435 | .............*****.....*.***X | 248.048 | | 190.903 | ..............*****......*.***X | 267.338 | | 266.495 | .....................*.***X | 285.649 | | 273.882 | .....................X | 273.882 | | 243.008 | ...................****.*........*****.....***X | 418.785 | | 347.678 | ...........................X | 347.678 | | 295.386 | .......................****............*...****X | 479.226 | | 300.444 | .......................***X | 300.444 | | 279.112 | ......................X | 279.112 | | 180.035 | .............*****.X | 187.404 | | 164.495 | ............****..*..............*X | 356.688 | | 356.688 | ............................**X | 356.688 | | 317.695 | .........................X | 317.695 | | 317.695 | .........................**X | 317.695 | | 317.695 | .........................*....*.*****.*..***X | 418.073 | | 387.527 | ...............................X | 387.527 | | 271.278 | .....................****X | 271.278 | | 181.778 | ..............*****X | 181.778 | | 177.779 | .............*.****.........*...****X | 326.129 | | 314.129 | .........................X | 314.129 | | 306.727 | ........................*****.....................****X | 560.883 | | 517.687 | ..........................................X | 517.687 | | 353.399 | ............................****X | 353.399 | | 246.168 | ...................X | 246.168 | | 226.562 | .................***X | 226.562 | | 226.562 | .................*.....******.........*****............*....****X | 587.201 | | 586.946 | ...............................................*.****X | 597.839 | | 456.655 | ....................................X | 456.655 | | 456.655 | ....................................****.****X | 462.295 | | 433.101 | ...................................X | 433.101 | | 314.691 | .........................*****........*****.......*..****X | 524.974 | | 423.812 | ..................................*****X | 423.812 | | 324.896 | ..........................*X | 324.896 | | 324.896 | ..........................**X | 324.896 | | 299.189 | .......................X | 299.189 | | 298.729 | .......................*****....X | 337.431 | | 333.905 | ..........................***X | 333.905 | | 310.157 | ........................*..****X | 327.618 | | 296.415 | .......................X | 296.415 | | 266.667 | .....................*****..*...*****.....*.....X | 450.893 | | 450.893 | ....................................***X | 450.893 | | 369.818 | .............................X | 369.818 | | 278.583 | ......................***X | 278.583 | | 259.694 | ....................X | 259.694 | | 187.462 | ..............*****.*.....*****....*****.X | 315.595 | | 258.142 | ....................*****.X | 269.213 | | 267.525 | .....................****X | 267.525 | | 218.259 | .................****X | 218.259 | | 186.604 | ..............X | 186.604 | | 162.409 | ............*****..*.......*****..........*...*****......****X | 497.909 | | 453.997 | ....................................X | 453.997 | | 415.493 | .................................****X | 415.493 | | 282.289 | ......................****X | 282.289 | | 243.905 | ...................*****X | 245.116 | | 196.921 | ...............*****...*......*....***X | 348.987 | | 331.080 | ..........................*.****X | 342.483 | | 311.845 | ........................X | 311.845 | | 237.119 | ..................*****...*****...*.........*****X | 412.909 | | 410.309 | .................................****X | 410.309 | | 291.986 | .......................X | 291.986 | | 270.568 | .....................*****........*****..*....*....****......*****X | 561.083 | | 439.026 | ...................................X | 439.026 | | 432.745 | ..................................****X | 432.745 | | 389.914 | ...............................*****X | 394.834 | | 320.362 | .........................****X | 320.362 | | 282.545 | ......................X | 282.545 | | 275.543 | .....................*****.........******.....****X | 434.072 | | 394.689 | ...............................X | 394.689 | | 379.682 | ..............................X | 379.682 | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 08:21:20,213 | test | INFO | pool-23-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******...........................................................*****...............***X | 179.184 | | 143.554 | ..........................................................X | 143.554 | | 103.747 | .........................................****...............****X | 138.420 | | 103.274 | .........................................****....X | 111.978 | | 105.691 | ..........................................*****...............................***X | 179.279 | | 158.854 | ................................................................X | 158.854 | | 151.659 | .............................................................****..X | 156.790 | | 151.702 | .............................................................**X | 151.702 | | 144.472 | ..........................................................*...****.X | 153.562 | | 142.070 | .........................................................****..............*.....***X | 188.357 | | 142.405 | .........................................................X | 142.405 | | 123.580 | ..................................................*****.......................****........*..................***X | 241.513 | | 192.940 | ..............................................................................X | 192.940 | | 164.437 | ...................................................................****.*..........***X | 192.138 | | 171.552 | ......................................................................X | 171.552 | | 151.511 | .............................................................***X | 151.511 | | 141.413 | .........................................................X | 141.413 | | 135.269 | .......................................................****X | 135.269 | | 75.189 | ..............................******.****.*.......****X | 97.081 | | 97.007 | .......................................****X | 97.007 | | 94.798 | ......................................X | 94.798 | | 68.778 | ...........................**X | 68.778 | | 68.778 | ...........................*X | 68.778 | | 62.086 | ........................X | 62.086 | | 59.289 | .......................*****........****X | 78.815 | | 76.013 | ..............................X | 76.013 | | 73.291 | .............................*****........****X | 92.294 | | 81.490 | ................................X | 81.490 | | 53.893 | .....................*****....*X | 63.019 | | 63.019 | .........................***X | 63.019 | | 46.384 | ..................*****................*..........****X | 109.400 | | 64.193 | .........................X | 64.193 | | 57.299 | ......................****X | 57.299 | | 52.198 | ....................X | 52.198 | | 51.083 | ....................*****...****X | 59.993 | | 51.990 | ....................X | 51.990 | | 21.007 | .......****X | 21.007 | | 14.607 | .....X | 14.607 | | 2.700 | ***X | 2.700 | | 1.800 | X | 1.800 | | 1.500 | *****..................*...*********X | 52.301 | | 44.201 | .................X | 44.201 | | 1.200 | *****.****X | 3.401 | | 2.001 | X | 2.001 | | 0.800 | ***************X | 2.000 | | 1.400 | X | 1.400 | | 0.600 | *****************............................*****..........*......****X | 106.981 | | 42.920 | ................X | 42.920 | | 1.800 | *****.............*.........*****...............*****X | 90.478 | | 58.709 | .......................*****.....*...*****.*****X | 81.286 | | 80.605 | ................................*****....*...****X | 97.886 | | 90.201 | ....................................X | 90.201 | | 82.704 | .................................***X | 82.704 | | 79.802 | ................................X | 79.802 | | 77.704 | ...............................****X | 77.704 | | 61.904 | ........................X | 61.904 | | 1.300 | *****.****X | 2.901 | | 2.300 | X | 2.300 | | 0.399 | ******.*****.......*****..........*......****X | 59.506 | | 50.906 | ....................X | 50.906 | | 20.299 | .......****X | 20.299 | | 14.101 | ....X | 14.101 | | 13.501 | ....*****................****X | 51.904 | | 32.602 | ............X | 32.602 | | 27.202 | ..........X | 27.202 | | 18.797 | ......***X | 18.797 | | 7.801 | ..X | 7.801 | | 1.600 | *****......*..*****................*...........***X | 84.889 | | 81.787 | ................................X | 81.787 | | 76.706 | ..............................****X | 76.706 | | 62.499 | ........................X | 62.499 | | 2.000 | ****X | 2.000 | | 1.600 | *****X | 1.600 | | 1.599 | ***********.....*...*****........................*****.*.........*****.....*..****X | 120.019 | | 64.324 | .........................X | 64.324 | | 26.198 | .........X | 26.198 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 08:21:20,282 | test | INFO | pool-23-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******............................................*****..........***X | 129.685 | | 116.163 | ...............................................X | 116.163 | | 84.463 | ..................................***X | 84.463 | | 77.407 | ...............................***X | 77.407 | | 74.590 | ..............................X | 74.590 | | 71.086 | ............................****...*****...X | 83.890 | | 79.386 | ................................****.............***X | 111.775 | | 111.004 | .............................................***X | 111.004 | | 105.977 | ...........................................X | 105.977 | | 105.570 | ...........................................****X | 106.478 | | 104.979 | ..........................................*****...................****X | 150.345 | | 146.676 | ............................................................*****X | 147.340 | | 146.935 | ............................................................****............................****X | 213.030 | | 150.849 | ..............................................................X | 150.849 | | 114.592 | ..............................................***X | 114.592 | | 102.668 | .........................................*****..........................................................****X | 239.106 | | 152.567 | ..............................................................*****....X | 160.411 | | 155.082 | ...............................................................****.........*..................****X | 217.927 | | 139.515 | .........................................................X | 139.515 | | 68.697 | ...........................****.......*.....****X | 96.396 | | 93.785 | ......................................*****X | 93.785 | | 91.792 | .....................................****X | 91.792 | | 81.612 | .................................X | 81.612 | | 78.609 | ...............................****X | 78.609 | | 58.492 | .......................******.................****X | 99.509 | | 89.703 | ....................................X | 89.703 | | 79.084 | ................................**********X | 80.608 | | 60.684 | ........................*****....*****X | 71.702 | | 65.916 | ..........................X | 65.916 | | 60.802 | ........................***X | 60.802 | | 58.101 | .......................X | 58.101 | | 54.678 | .....................****X | 54.678 | | 53.480 | .....................X | 53.480 | | 40.196 | ...............*****.........*****X | 59.821 | | 0.900 | **********........*................................*****.......****X | 113.003 | | 90.805 | ....................................X | 90.805 | | 73.090 | .............................****X | 73.090 | | 15.101 | .....*****X | 15.101 | | 1.000 | ******..............................*****....*..****X | 87.471 | | 79.272 | ................................X | 79.272 | | 76.607 | ...............................*****.....*****.X | 90.900 | | 89.605 | ....................................****X | 89.605 | | 81.405 | .................................X | 81.405 | | 52.192 | ....................****X | 52.192 | | 37.292 | ..............*****X | 37.292 | | 2.801 | ****X | 2.801 | | 2.001 | X | 2.001 | | 1.200 | *****........*.....*****...........****X | 58.504 | | 49.898 | ...................X | 49.898 | | 20.312 | .......X | 20.312 | | 7.799 | ..***X | 7.799 | | 7.598 | ..X | 7.598 | | 0.600 | ****X | 0.600 | | 0.200 | *****.............*........****X | 50.991 | | 49.704 | ...................*****X | 49.704 | | 23.810 | ........X | 23.810 | | 3.901 | ***X | 3.901 | | 2.601 | X | 2.601 | | 1.801 | X | 1.801 | | 1.000 | ***X | 1.000 | | 0.400 | X | 0.400 | | 0.200 | *****..*.....*****....................................*****X | 103.578 | | 59.403 | .......................*****.*****X | 62.005 | | 22.099 | ........******......................................****X | 114.392 | | 97.295 | .......................................X | 97.295 | | 91.014 | .....................................X | 91.014 | | 81.208 | ................................****....*....****X | 98.992 | | 77.001 | ...............................X | 77.001 | | 66.198 | ..........................****X | 66.198 | | 24.707 | .........X | 24.707 | | 0.400 | *****.****X | 2.800 | | 1.200 | X | 1.200 | | 0.000 | ****************..................*......*.......***X | 74.889 | | 52.304 | ....................X | 52.304 | | 44.102 | .................*****.....................****X | 95.562 | | 84.364 | ..................................X | 84.364 | | 65.512 | ..........................****X | 65.512 | | 26.517 | ..........X | 26.517 | | 2.300 | *****..*......*****.................*X | 60.498 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 08:21:20,319 | test | WARNING | Thread-73654 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DUdNE3-H4s-56-533000 update failed keys from task: [u'test_collections-1235', u'test_collections-1379', u'test_collections-1183', u'test_collections-1394', u'test_collections-1266', u'test_collections-1244', u'test_collections-1223', u'test_collections-1201'] 2021-05-17 08:21:20,323 | test | WARNING | Thread-73671 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:zngj-56-534000 update failed keys from task: [u'test_collections-1190', u'test_collections-982', u'test_collections-1269', u'test_collections-1348', u'test_collections-1161', u'test_collections-1085', u'test_collections-1284', u'test_collections-1068', u'test_collections-1115', u'test_collections-1314', u'test_collections-1217', u'test_collections-1150', u'test_collections-1031', u'test_collections-1275', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 08:21:20,325 | test | WARNING | Thread-73674 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-1190', u'test_collections-1270', u'test_collections-982', u'test_collections-1269', u'test_collections-1401', u'test_collections-1348', u'test_collections-1429', u'test_collections-1161', u'test_collections-1085', u'test_collections-1360', u'test_collections-1284', u'test_collections-1068', u'test_collections-1189', u'test_collections-1115', u'test_collections-1314', u'test_collections-1150', u'test_collections-1031', u'test_collections-1394', u'test_collections-1230', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 08:21:20,326 | test | WARNING | Thread-73678 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F9k7JEZ1OQXr82go-56-534000 update failed keys from task: [u'test_collections-982', u'test_collections-1269', u'test_collections-1161', u'test_collections-1284', u'test_collections-1068', u'test_collections-1257', u'test_collections-1314', u'test_collections-1217', u'test_collections-1417', u'test_collections-1150', u'test_collections-1031', u'test_collections-1394', u'test_collections-1275', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 08:21:20,329 | test | WARNING | Thread-73688 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:eeZ-23-758000:ADtP-23-863000 create failed keys from task: [u'test_collections-2313', u'test_collections-2283'] 2021-05-17 08:21:20,329 | test | WARNING | Thread-73689 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:eeZ-23-758000:VSuGB-23-863000 create failed keys from task: [u'test_collections-1124'] 2021-05-17 08:21:20,332 | test | WARNING | Thread-73698 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Dk8ik4-56-543000:nnNMXk_zN2z0bW29ib-56-544000 update failed keys from task: [u'test_collections-1106', u'test_collections-1284', u'test_collections-1080', u'test_collections-1257', u'test_collections-1314', u'test_collections-1217', u'test_collections-1096', u'test_collections-1110', u'test_collections-1056', u'test_collections-1430', u'test_collections-1013'] 2021-05-17 08:21:20,334 | test | WARNING | Thread-73705 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Dk8ik4-56-543000:N9qSlfFZuby%-56-545000 update failed keys from task: [u'test_collections-1418'] 2021-05-17 08:21:20,341 | test | WARNING | Thread-73722 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1dnqgOucFPG7KRD9S-22-822000:QLIfDTjdUOLJ15JV-22-936000 create failed keys from task: [u'test_collections-1957'] 2021-05-17 08:21:20,342 | test | WARNING | Thread-73728 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:LWCmMGG%khkO-56-537000 update failed keys from task: [u'test_collections-1106', u'test_collections-1284', u'test_collections-1068', u'test_collections-1080', u'test_collections-1257', u'test_collections-1314', u'test_collections-1217', u'test_collections-1074', u'test_collections-1096', u'test_collections-1252', u'test_collections-1110', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 08:21:20,361 | test | WARNING | Thread-73705 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Dk8ik4-56-543000:N9qSlfFZuby%-56-545000 delete failed keys from task: [u'test_collections-939', u'test_collections-869', u'test_collections-812', u'test_collections-911', u'test_collections-857', u'test_collections-954', u'test_collections-830', u'test_collections-785', u'test_collections-780'] 2021-05-17 08:21:20,384 | test | WARNING | Thread-73871 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:OE8-i7V1-20-871000:N-20-990000 create failed keys from task: [u'test_collections-363'] 2021-05-17 08:21:20,385 | test | WARNING | Thread-73875 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:W-20-282000:VgCXaNnuZTXP-20-425000 create failed keys from task: [u'test_collections-1634'] 2021-05-17 08:21:20,387 | test | WARNING | Thread-73876 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:W-20-282000:CJ%nyboOP7Ejd-20-425000 create failed keys from task: [u'test_collections-1634'] 2021-05-17 08:21:20,387 | test | WARNING | Thread-73877 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Hlpnq-20-563000:JP2k0NDZ7fdE900-20-681000 create failed keys from task: [u'test_collections-1634'] 2021-05-17 08:21:20,387 | test | WARNING | Thread-73878 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Hlpnq-20-563000:4g9wde2a-24-85000 create failed keys from task: [u'test_collections-1634'] 2021-05-17 08:21:20,421 | test | WARNING | Thread-74015 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bF-24-444000:S-24-582000 update failed keys from task: [u'test_collections-1190', u'test_collections-1103', u'test_collections-1106', u'test_collections-1109', u'test_collections-1307', u'test_collections-1183', u'test_collections-1143', u'test_collections-1342', u'test_collections-1189', u'test_collections-1059', u'test_collections-1138', u'test_collections-1019', u'test_collections-1172', u'test_collections-1096', u'test_collections-1099', u'test_collections-1297', u'test_collections-1056', u'test_collections-1013', u'test_collections-1093', u'test_collections-1204', u'test_collections-1241', u'test_collections-1000', u'test_collections-1121', u'test_collections-1045', u'test_collections-1394', u'test_collections-1155'] 2021-05-17 08:21:34,740 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 08:22:49,670 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 08:22:49,671 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 08:23:08,835 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 08:23:08,836 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 08:23:24,375 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 08:23:24,375 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 08:24:20,473 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1270000935 seconds to complete 2021-05-17 08:24:20,489 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3227.880s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 21 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_21 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_22 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_22', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 22, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 08:24:29,960 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #22 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 08:24:30,625 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #22 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 08:24:31,052 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc77da9 type=client open_count=1 channel=[id: 0x0eb04b31, 0.0.0.0/0.0.0.0:58402] timeout=300.0>) 2021-05-17 08:24:51,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc77daa type=client open_count=1 channel=[id: 0x2ae9b169, 0.0.0.0/0.0.0.0:54446] timeout=300.0>) 2021-05-17 08:24:53,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc77dab type=client open_count=1 channel=[id: 0x3bab3b4c, 0.0.0.0/0.0.0.0:34522] timeout=300.0>) 2021-05-17 08:24:55,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xc77dac type=client open_count=1 channel=[id: 0x5d76c447, 0.0.0.0/0.0.0.0:57360] timeout=300.0>) 2021-05-17 08:25:19,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xc77dad type=client open_count=1 channel=[id: 0x98525feb, 0.0.0.0/0.0.0.0:47852] timeout=300.0>) 2021-05-17 08:25:21,859 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xc77dae type=client open_count=1 channel=[id: 0x71ad752f, 0.0.0.0/0.0.0.0:46666] timeout=300.0>) 2021-05-17 08:25:23,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xc77daf type=client open_count=1 channel=[id: 0x35e75aa7, 0.0.0.0/0.0.0.0:54877] timeout=300.0>) 2021-05-17 08:25:47,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xc77db0 type=client open_count=1 channel=[id: 0x377b6f6e, 0.0.0.0/0.0.0.0:59327] timeout=300.0>) 2021-05-17 08:25:49,829 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xc77db1 type=client open_count=1 channel=[id: 0xdef30d0e, 0.0.0.0/0.0.0.0:56055] timeout=300.0>) 2021-05-17 08:25:51,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xc77db2 type=client open_count=1 channel=[id: 0xe9854371, 0.0.0.0/0.0.0.0:41000] timeout=300.0>) 2021-05-17 08:26:15,867 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xc77db3 type=client open_count=1 channel=[id: 0xb7a8f47d, 0.0.0.0/0.0.0.0:51433] timeout=300.0>) 2021-05-17 08:26:17,884 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xc77db4 type=client open_count=1 channel=[id: 0x1695c919, 0.0.0.0/0.0.0.0:53674] timeout=300.0>) 2021-05-17 08:26:19,901 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xc77db5 type=client open_count=1 channel=[id: 0xa3ff3e06, 0.0.0.0/0.0.0.0:57316] timeout=300.0>) 2021-05-17 08:26:43,822 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xc77db6 type=client open_count=1 channel=[id: 0xcf6137e0, 0.0.0.0/0.0.0.0:47210] timeout=300.0>) 2021-05-17 08:26:45,839 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xc77db7 type=client open_count=1 channel=[id: 0xd3e15dc2, 0.0.0.0/0.0.0.0:34175] timeout=300.0>) 2021-05-17 08:26:47,857 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 08:27:04,943 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 08:27:18,680 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 08:27:18,703 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #22 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 08:27:18,742 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #22 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 08:27:31,696 | test | INFO | pool-24-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.6318210607 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 08:27:31,720 | test | INFO | pool-24-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 08:27:41,793 | test | INFO | pool-24-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 08:27:41,821 | test | INFO | pool-24-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1260001659 sec 2021-05-17 08:27:42,066 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 08:28:30,704 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 08:28:31,032 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 589698024 | 87147178 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 644539408 | 115109393 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 901527256 | 44062187 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 08:28:31,032 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 08:29:17,500 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 08:29:17,516 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 08:45:16,288 | test | INFO | pool-24-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****.................................*...................................................................****X | 86.998 | | 69.598 | ..............................................................................X | 69.598 | | 0.000 | ****************************...****X | 2.999 | | 0.300 | *****...******.......*...****X | 11.500 | | 5.208 | ....****X | 5.208 | | 4.908 | ....X | 4.908 | | 1.500 | ****X | 1.500 | | 0.000 | *****************************.******...*.........****X | 11.999 | | 10.500 | ...........*****X | 10.500 | | 0.000 | ******************.*****.......****X | 7.790 | | 5.791 | .....X | 5.791 | | 4.595 | ....****X | 4.595 | | 3.695 | ...X | 3.695 | | 0.000 | *****......*****X | 5.895 | | 4.596 | ....*****X | 4.596 | | 0.400 | *****.*.****X | 2.597 | | 0.798 | X | 0.798 | | 0.000 | ******.*****........******..*****X | 9.894 | | 0.000 | ***********************......****X | 5.996 | | 0.000 | *****************......*****X | 5.996 | | 5.995 | .....****X | 5.996 | | 5.196 | ....X | 5.196 | | 3.000 | ..****X | 3.000 | | 2.999 | ..******..*.......****X | 10.496 | | 0.000 | ****************************............****X | 10.496 | | 0.000 | ************.******.......****X | 7.494 | | 0.000 | ******...*****X | 2.999 | | 1.000 | ****X | 1.000 | | 0.800 | X | 0.800 | | 0.500 | *****X | 0.500 | | 0.000 | *****...*..***X | 4.500 | | 4.499 | ....X | 4.499 | | 4.499 | ....****X | 4.499 | | 1.500 | *****X | 1.500 | | 1.500 | ******.....****X | 5.999 | | 5.799 | .....X | 5.799 | | 2.999 | ..**********X | 2.999 | | 0.000 | ******...*.*****....****X | 7.801 | | 1.802 | .X | 1.802 | | 0.000 | ****************************...*.****X | 4.298 | | 3.799 | ...X | 3.799 | | 3.199 | ..***X | 3.199 | | 2.899 | ..*****X | 2.999 | | 2.999 | ..X | 2.999 | | 2.999 | ..****X | 2.999 | | 0.000 | *****.*****X | 1.499 | | 0.500 | X | 0.500 | | 0.000 | *****.****X | 1.499 | | 0.400 | X | 0.400 | | 0.000 | *****..*.****X | 3.003 | | 1.501 | ****X | 1.501 | | 0.801 | X | 0.801 | | 0.000 | *****..*.***X | 3.002 | | 2.203 | .X | 2.203 | | 0.000 | *********************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 08:45:16,440 | test | INFO | pool-24-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****..............................................................................****X | 15.000 | | 12.000 | ..............................................................X | 12.000 | | 0.000 | **********************************.......****X | 1.500 | | 0.000 | ******.......................................................................****X | 13.500 | | 12.600 | .................................................................X | 12.600 | | 0.000 | ***********...*........*****......*****X | 3.300 | | 2.000 | .........*****..*...****X | 3.000 | | 2.400 | ...........X | 2.400 | | 0.000 | ***********...............................*****.*.......****X | 7.500 | | 1.500 | ......X | 1.500 | | 0.000 | *****...............*****X | 2.996 | | 1.498 | ......******.....................................................*****X | 11.490 | | 3.495 | .................****X | 3.495 | | 1.195 | .....X | 1.195 | | 0.000 | ****************....*...........****X | 2.997 | | 0.797 | ...X | 0.797 | | 0.000 | *****...........*...........................................*****..*****X | 10.694 | | 5.796 | .............................*****..............*...................................****X | 15.086 | | 8.389 | ...........................................X | 8.389 | | 5.996 | ..............................****X | 5.996 | | 1.596 | .......X | 1.596 | | 0.000 | *****...............*****X | 2.998 | | 2.998 | ..............*****X | 2.998 | | 1.499 | ......*****........*****X | 2.998 | | 2.198 | ..........X | 2.198 | | 0.000 | **********..........................******..........................................................................****X | 18.988 | | 1.499 | ......******..............*****....*****X | 4.897 | | 4.497 | ......................X | 4.497 | | 2.999 | ..............*****........****X | 4.499 | | 4.498 | ......................*****X | 4.498 | | 0.000 | ****************..*.............................****X | 5.998 | | 2.399 | ...........X | 2.399 | | 1.499 | ......****X | 1.499 | | 0.000 | ******...............................******......*..****X | 7.494 | | 1.500 | ......*****.....*...................****X | 5.998 | | 0.000 | *****************..*.....*****................******................****X | 7.499 | | 7.499 | ......................................X | 7.499 | | 7.498 | ......................................*****................*****X | 10.498 | | 4.498 | ......................*****X | 4.498 | | 1.500 | ......*****X | 1.500 | | 0.000 | *****...........******.........****X | 3.801 | | 0.901 | ...X | 0.901 | | 0.000 | ***********.......................****X | 4.499 | | 2.300 | ...........X | 2.300 | | 1.499 | ......*****................................*****..........*....****X | 10.195 | | 3.100 | ...............X | 3.100 | | 0.300 | ****X | 0.300 | | 0.000 | ******...............****X | 2.999 | | 2.299 | ...........X | 2.299 | | 1.899 | .........X | 1.899 | | 1.500 | ......***X | 1.500 | | 0.800 | ...X | 0.800 | | 0.000 | **********...*............****X | 2.999 | | 1.999 | .........X | 1.999 | | 1.501 | ......****...............*.................****X | 7.507 | | 6.307 | ................................X | 6.307 | | 1.501 | ......**********........*****X | 3.002 | | 0.000 | **********************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 08:45:17,921 | test | INFO | pool-24-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****..............................*..............................................*****.....*...................****X | 847.086 | | 673.955 | ..............................................................................****X | 673.955 | | 647.217 | ...........................................................................X | 647.217 | | 623.088 | ........................................................................*****.......******...*****.....*.****X | 758.465 | | 621.781 | ........................................................................*****X | 621.781 | | 549.932 | ...............................................................*****X | 549.932 | | 461.900 | .....................................................****X | 461.900 | | 456.221 | ....................................................******..*.........***********......*****X | 594.623 | | 553.100 | ................................................................****X | 553.100 | | 528.480 | .............................................................******........*****X | 595.764 | | 488.270 | ........................................................****X | 488.270 | | 469.909 | ......................................................*...******.....****X | 537.798 | | 536.832 | ..............................................................*......****X | 588.553 | | 518.252 | ............................................................*****X | 518.252 | | 487.634 | ........................................................****X | 487.634 | | 418.487 | ................................................*****X | 418.487 | | 414.861 | ...............................................X | 414.861 | | 407.730 | ...............................................***********......................****X | 600.222 | | 573.209 | ..................................................................X | 573.209 | | 564.150 | .................................................................****X | 564.150 | | 503.508 | ..........................................................X | 503.508 | | 492.146 | .........................................................*****X | 499.575 | | 485.194 | ........................................................*****..*****X | 504.815 | | 469.026 | ......................................................*****...*.*****X | 504.927 | | 498.178 | .........................................................****X | 498.178 | | 494.656 | .........................................................*****X | 494.656 | | 485.459 | ........................................................****X | 485.459 | | 478.533 | .......................................................X | 478.533 | | 421.279 | ................................................*****.........*****X | 494.704 | | 493.194 | .........................................................X | 493.194 | | 460.489 | .....................................................***X | 460.489 | | 451.479 | ....................................................*.....*****.*...****X | 529.264 | | 519.117 | ............................................................*****.*****X | 531.734 | | 515.557 | ...........................................................*****........*****X | 582.618 | | 516.291 | ...........................................................*****X | 516.291 | | 510.898 | ...........................................................*****X | 510.898 | | 454.865 | ....................................................*****...*.....*****....*****X | 552.802 | | 540.157 | ..............................................................*****.*****X | 545.497 | | 439.409 | ..................................................****X | 439.409 | | 435.227 | ..................................................*..*****.............*****X | 564.501 | | 552.240 | ................................................................****X | 552.240 | | 524.650 | ............................................................X | 524.650 | | 494.936 | .........................................................****X | 494.936 | | 492.254 | .........................................................******..****X | 514.075 | | 468.214 | ......................................................*..*****..*.......****X | 560.327 | | 471.635 | ......................................................******..........*****X | 551.015 | | 442.490 | ...................................................*****...******......*****X | 517.478 | | 504.604 | ..........................................................****X | 504.604 | | 497.696 | .........................................................X | 497.696 | | 471.048 | ......................................................*****...*****X | 492.161 | | 481.061 | .......................................................*....****X | 508.409 | | 460.649 | .....................................................*****X | 460.649 | | 445.582 | ...................................................*****...........*****X | 538.151 | | 464.567 | .....................................................****X | 464.567 | | 463.667 | .....................................................*....X | 494.940 | | 485.828 | ........................................................***X | 485.828 | | 479.003 | .......................................................X | 479.003 | | 460.513 | .....................................................*****..*.....****X | 519.295 | | 440.305 | ..................................................*****.....X | 482.631 | | 480.287 | .......................................................*****.........****X | 558.546 | | 525.500 | .............................................................X | 525.500 | | 505.032 | ..........................................................****X | 505.032 | | 486.324 | ........................................................X | 486.324 | | 449.119 | ....................................................*****.....*****.X | 500.932 | | 471.186 | ......................................................*****..*.....***X | 528.532 | | 516.110 | ...........................................................X | 516.110 | | 470.395 | ......................................................*****..*.......****..X | 566.562 | | 556.847 | ................................................................****X | 556.847 | | 545.796 | ...............................................................X | 545.796 | | 515.099 | ...........................................................****X | 515.099 | | 502.286 | ..........................................................****X | 502.286 | | 470.650 | ......................................................****X | 470.650 | | 447.675 | ...................................................X | 447.675 | | 308.436 | ...................................*****....*......................*****.................****X | 671.463 | | 626.257 | ........................................................................*..****X | 643.358 | | 464.189 | .....................................................X | 464.189 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 08:45:18,196 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 62.9689505175 | 6437847040 | 4325212160 | 2418905088 / 7511588864 | 101194 / 205400 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 76.3253945768 | 6437847040 | 4226244608 | 2480291840 / 7511588864 | 100200 / 201600 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 61.695 | 6437847040 | 4266299392 | 2429489152 / 7511588864 | 100600 / 199793 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 65.1276144691 | 6437847040 | 3897257984 | 3003187200 / 9530753024 | 101800 / 202597 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 63.1602626054 | 6437847040 | 4263813120 | 2445479936 / 7511588864 | 102200 / 202596 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 08:45:31,654 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 08:46:58,357 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164630584 | 244968471 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162245712 | 285608859 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 445445096 | 434961263 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 08:46:58,358 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #22 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 08:47:08,520 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 08:47:08,522 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 08:47:08,522 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 08:47:50,437 | test | INFO | pool-24-thread-16 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 41.7230000496 seconds to complete 2021-05-17 08:47:50,441 | test | INFO | pool-24-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 08:49:55,506 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0609998703003 seconds 2021-05-17 08:49:55,506 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 08:49:55,507 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 08:50:02,411 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 09:02:25,269 | test | INFO | pool-24-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.***********X | 1.801 | | 0.200 | *****.*.****X | 3.401 | | 3.302 | .****X | 3.302 | | 3.002 | .X | 3.002 | | 2.701 | ****X | 2.701 | | 1.300 | ******.****X | 1.501 | | 1.401 | X | 1.401 | | 1.300 | *****X | 1.300 | | 0.800 | ******...**X | 4.704 | | 4.704 | ..*X | 4.704 | | 3.104 | .X | 3.104 | | 2.000 | **X | 2.000 | | 2.000 | **............*.....****.......*......................****X | 64.201 | | 58.208 | .........................................X | 58.208 | | 53.203 | .....................................****X | 53.203 | | 52.898 | .....................................*...*****..X | 59.002 | | 58.702 | .........................................*****...*..*****..X | 67.401 | | 65.399 | ..............................................****X | 65.399 | | 61.192 | ...........................................X | 61.192 | | 59.804 | ..........................................*****..............*......****X | 86.502 | | 82.500 | ...........................................................X | 82.500 | | 76.816 | .......................................................*****....*..****X | 86.009 | | 85.303 | .............................................................X | 85.303 | | 76.476 | ......................................................*****.................*.....***X | 105.433 | | 98.033 | ......................................................................*..******.................****X | 123.209 | | 119.707 | ......................................................................................X | 119.707 | | 118.717 | .....................................................................................****X | 118.717 | | 102.059 | .........................................................................X | 102.059 | | 94.357 | ....................................................................****X | 94.357 | | 82.215 | ...........................................................X | 82.215 | | 79.718 | .........................................................*****........*....*****........X | 106.904 | | 106.199 | ............................................................................****X | 106.199 | | 84.303 | ............................................................X | 84.303 | | 83.405 | ............................................................****X | 83.405 | | 83.109 | ...........................................................X | 83.109 | | 79.504 | .........................................................*****.......*..*****....X | 97.990 | | 89.792 | ................................................................*****.......*.......****X | 109.221 | | 106.713 | .............................................................................*X | 106.713 | | 102.595 | ..........................................................................****.........*.*****.......*........****X | 136.581 | | 134.072 | .................................................................................................X | 134.072 | | 124.615 | ..........................................................................................****X | 124.615 | | 106.224 | ............................................................................*....*****.......*****X | 121.403 | | 108.799 | ..............................................................................X | 108.799 | | 102.979 | ..........................................................................*****.*...****X | 107.931 | | 73.627 | ....................................................X | 73.627 | | 66.394 | ...............................................****X | 66.394 | | 3.001 | .*****X | 3.001 | | 2.600 | *****X | 2.600 | | 0.700 | ******..*****X | 3.701 | | 1.900 | X | 1.900 | | 1.200 | *****..*****X | 3.501 | | 0.400 | ******..****X | 3.200 | | 1.700 | X | 1.700 | | 1.600 | ***********X | 2.699 | | 1.200 | ******...****X | 4.500 | | 4.100 | ..X | 4.100 | | 2.400 | *****X | 2.400 | | 1.500 | ************.*****X | 3.001 | | 3.000 | .*****X | 3.000 | | 1.700 | *****.X | 3.100 | | 3.100 | .****X | 3.100 | | 2.200 | ******..******X | 4.300 | | 2.700 | ****X | 2.700 | | 2.100 | ***********X | 2.400 | | 2.200 | **********X | 2.200 | | 1.700 | **********X | 2.400 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:02:26,051 | test | INFO | pool-24-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******.......*...****............*.............*........................****..........*........****X | 212.312 | | 211.280 | ............................................................................*.........****X | 236.362 | | 213.254 | ............................................................................X | 213.254 | | 189.122 | ....................................................................*****..........*****......*....*****..*****X | 249.829 | | 224.243 | ................................................................................*****...*.....*.........****..X | 273.627 | | 261.311 | ..............................................................................................****X | 261.311 | | 227.309 | ..................................................................................X | 227.309 | | 221.923 | ................................................................................****X | 221.923 | | 198.210 | .......................................................................X | 198.210 | | 190.799 | ....................................................................****X | 190.799 | | 164.613 | ...........................................................******..........*****X | 191.997 | | 172.694 | ..............................................................*****X | 172.694 | | 100.608 | ...................................X | 100.608 | | 82.689 | .............................****X | 82.689 | | 79.802 | ............................X | 79.802 | | 77.399 | ...........................****X | 77.399 | | 67.494 | .......................X | 67.494 | | 61.104 | .....................****X | 61.104 | | 56.811 | ...................*.***X | 59.102 | | 31.607 | ..........X | 31.607 | | 5.502 | .****X | 5.502 | | 2.902 | X | 2.902 | | 1.000 | **********X | 1.499 | | 0.799 | X | 0.799 | | 0.200 | ****X | 0.300 | | 0.300 | ***********.*****X | 3.798 | | 2.000 | X | 2.000 | | 1.100 | ****X | 1.100 | | 0.500 | *X | 0.500 | | 0.000 | ************.***X | 3.500 | | 3.200 | X | 3.200 | | 2.000 | ****X | 2.000 | | 0.700 | X | 0.700 | | 0.200 | *****X | 0.200 | | 0.200 | *****X | 0.300 | | 0.200 | ***********.*****X | 4.200 | | 3.200 | *X | 3.200 | | 2.500 | ***X | 2.500 | | 1.500 | X | 1.500 | | 0.800 | ************.*****X | 2.800 | | 0.500 | ***********.*****X | 4.102 | | 2.802 | X | 2.802 | | 1.600 | ****X | 1.600 | | 1.400 | ************X | 1.600 | | 0.300 | ******.****X | 5.101 | | 0.700 | ******.*****X | 4.501 | | 0.700 | ******************..*****X | 5.599 | | 1.200 | ******..*****X | 6.800 | | 0.700 | ******.*****X | 3.600 | | 2.300 | *****X | 2.300 | | 1.600 | ******..*****X | 5.800 | | 0.600 | ***********X | 1.800 | | 1.800 | ******.*****X | 2.800 | | 2.100 | ******.*****X | 3.100 | | 2.400 | **********X | 2.700 | | 1.400 | *****X | 1.400 | | 1.200 | ******.*****X | 3.100 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:02:26,783 | test | INFO | pool-24-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................*................****X | 906.741 | | 209.121 | ..............******...........................................******..........................................****X | 1326.945 | | 1136.474 | ....................................................................................X | 1136.474 | | 585.542 | ...........................................******....**X | 648.761 | | 648.761 | ...............................................*X | 648.761 | | 618.079 | .............................................X | 618.079 | | 598.770 | ............................................**********X | 607.614 | | 343.349 | ........................******........................**X | 663.328 | | 663.328 | ................................................*X | 663.328 | | 616.581 | .............................................*.*..***X | 656.684 | | 492.002 | ....................................X | 492.002 | | 415.812 | ..............................****X | 415.812 | | 407.296 | .............................*.****X | 419.433 | | 406.218 | .............................X | 406.218 | | 392.498 | ............................****X | 392.498 | | 294.139 | .....................X | 294.139 | | 271.116 | ...................****....*..*..****........*..***X | 509.197 | | 509.197 | .....................................X | 509.197 | | 473.445 | ..................................X | 473.445 | | 460.634 | .................................****X | 460.634 | | 317.722 | ......................X | 317.722 | | 259.593 | ..................*****X | 259.964 | | 241.094 | .................****X | 241.094 | | 220.831 | ...............*...****...*X | 303.717 | | 294.657 | .....................****....*...****X | 397.039 | | 379.501 | ...........................X | 379.501 | | 367.882 | ..........................*****..........*...*X | 537.572 | | 537.572 | .......................................*X | 537.572 | | 537.572 | .......................................X | 537.572 | | 458.659 | .................................X | 458.659 | | 456.837 | .................................****X | 456.837 | | 450.022 | ................................X | 450.022 | | 418.725 | ..............................****X | 418.725 | | 396.024 | ............................*.****X | 410.618 | | 354.606 | .........................X | 354.606 | | 323.500 | .......................*****.........*...*****...*..****X | 555.846 | | 491.990 | ....................................*****X | 491.990 | | 461.325 | .................................****X | 461.325 | | 428.354 | ...............................X | 428.354 | | 376.721 | ...........................*****..X | 407.145 | | 403.663 | .............................*.****X | 415.563 | | 319.371 | .......................******....*****X | 378.272 | | 261.285 | ..................******....................****X | 527.772 | | 485.811 | ...................................X | 485.811 | | 422.251 | ..............................****X | 422.251 | | 412.317 | ..............................*****X | 422.347 | | 393.779 | ............................X | 393.779 | | 379.220 | ...........................*****....*.*****X | 444.312 | | 297.636 | .....................******.******....****X | 361.070 | | 361.070 | ..........................*.............******....****X | 596.234 | | 575.883 | ..........................................X | 575.883 | | 540.856 | .......................................******..*X | 568.369 | | 568.369 | .........................................***X | 568.369 | | 507.471 | .....................................**X | 507.471 | | 507.471 | .....................................*X | 507.471 | | 507.471 | .....................................*...*****X | 551.642 | | 494.099 | ....................................**X | 494.099 | | 494.099 | ....................................**....*.....X | 622.283 | | 622.283 | .............................................*X | 622.283 | | 622.283 | .............................................*X | 622.283 | | 618.717 | .............................................X | 618.717 | | 616.933 | .............................................****X | 616.933 | | 549.583 | ........................................***********X | 554.842 | | 460.577 | .................................******........*****X | 561.843 | | 493.594 | ....................................******........*****X | 608.843 | | 528.454 | ......................................******.*****X | 537.215 | | 526.838 | ......................................****X | 526.838 | | 525.239 | ......................................X | 525.239 | | 520.214 | ......................................****X | 520.214 | | 489.500 | ...................................******...*.......****X | 621.731 | | 581.931 | ..........................................X | 581.931 | | 524.510 | ......................................*****...*****X | 564.327 | | 519.686 | ......................................*****X | 519.686 | | 424.630 | ...............................*****...............******.******..*****X | 673.731 | +----------+----------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 09:02:27,068 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 09:02:27,140 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 09:02:37,710 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 09:02:38,032 | test | INFO | pool-24-thread-13 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.258346250646 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 42.4726254562 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 31.4565728286 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 59.0541848897 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 29.8683333333 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 09:02:38,099 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:02:43,178 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:02:47,959 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 09:02:48,289 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:02:53,459 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:02:58,644 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:03:03,742 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 2.01 2021-05-17 09:03:08,816 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 5.78 2021-05-17 09:03:13,921 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 8.27 2021-05-17 09:03:19,134 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 15.93 2021-05-17 09:03:24,262 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 21.19 2021-05-17 09:03:29,391 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 22.53 2021-05-17 09:03:34,493 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 25.93 2021-05-17 09:03:39,592 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 29.17 2021-05-17 09:03:44,826 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 35.24 2021-05-17 09:03:49,973 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 48.6 2021-05-17 09:03:53,857 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in bucket2:g-42-122000:u6lXOUhLtsRVgu3YgiJ-42-124000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:03:55,091 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 55.46 2021-05-17 09:04:00,180 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 58.35 2021-05-17 09:04:05,335 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.28 2021-05-17 09:04:05,801 | test | INFO | pool-24-thread-4 | [table_view:display:72] Keys failed in bucket2:RZ2sSWVCxUEaYB-42-703000:P-42-839000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2450 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:05,802 | test | INFO | pool-24-thread-12 | [table_view:display:72] Keys failed in bucket2:RZ2sSWVCxUEaYB-42-703000:I1%svD1B-42-840000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2450 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:05,802 | test | INFO | pool-24-thread-4 | [table_view:display:72] Keys failed in bucket2:RZ2sSWVCxUEaYB-42-703000:P-42-839000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2450 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:05,802 | test | INFO | pool-24-thread-12 | [table_view:display:72] Keys failed in bucket2:RZ2sSWVCxUEaYB-42-703000:I1%svD1B-42-840000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2450 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:09,740 | test | INFO | pool-24-thread-30 | [table_view:display:72] Keys failed in default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-67 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:09,805 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:V53XUL3Zts4gigdmiDD-2-138000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:10,451 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 81.74 2021-05-17 09:04:14,322 | test | INFO | pool-24-thread-3 | [table_view:display:72] Keys failed in default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1761 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:15,552 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 86.22 2021-05-17 09:04:18,877 | test | INFO | pool-24-thread-16 | [table_view:display:72] Keys failed in default:_default:7-42-140000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:19,744 | test | INFO | pool-24-thread-22 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:UCG-42-149000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:20,651 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: running, progress: 89.88 2021-05-17 09:04:21,638 | test | INFO | pool-24-thread-23 | [table_view:display:72] Keys failed in default:n1Iz4LJLjZWb6-46-129000:s-46-678000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:21,638 | test | INFO | pool-24-thread-23 | [table_view:display:72] Keys failed in default:n1Iz4LJLjZWb6-46-129000:s-46-678000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-830 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:22,586 | test | INFO | pool-24-thread-5 | [table_view:display:72] Keys failed in default:-SUuiHHJz-1-814000:pSgP_LfU%fUStgHUWC-1-943000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1132 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:23,960 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:V53XUL3Zts4gigdmiDD-2-138000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1576 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1554 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1458 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1547 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1591 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1475 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1542 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:24,046 | test | INFO | pool-24-thread-30 | [table_view:display:72] Keys failed in default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-208 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-304 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-326 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-247 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-236 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-357 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-369 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-279 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-202 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-398 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-363 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-294 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-273 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:24,441 | test | INFO | pool-24-thread-27 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:V53XUL3Zts4gigdmiDD-2-138000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-317 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-304 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-287 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-294 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:25,164 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:FsX_xjZauPHZAYE%-42-149000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:26,928 | test | INFO | pool-24-thread-3 | [table_view:display:72] Keys failed in default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1807 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1923 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1901 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1935 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1891 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:27,693 | test | INFO | pool-24-thread-1 | [table_view:display:72] Keys failed in default:9KnnReY1QD-47-523000:ly0DdlZAfv5_Pc-47-620000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-812 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-982 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:27,694 | test | INFO | pool-24-thread-1 | [table_view:display:72] Keys failed in default:9KnnReY1QD-47-523000:ly0DdlZAfv5_Pc-47-620000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-812 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-982 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:30,601 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:d-42-148000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:04:30,775 | test | INFO | pool-24-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 09:04:30,815 | test | INFO | pool-24-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 112.782999992 sec 2021-05-17 09:04:33,891 | test | INFO | pool-24-thread-20 | [table_view:display:72] Keys failed in default:pd7XdB%AI-42-147000:FsX_xjZauPHZAYE%-42-149000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-615 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-621 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-523 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-677 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-664 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-650 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:09:09,051 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 09:09:09,052 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 09:09:09,053 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 09:09:09,055 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 09:09:09,056 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 09:09:09,058 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 10 docs from 1000 to 1960 2021-05-17 09:09:55,279 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 09:09:55,280 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 09:09:55,282 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 13 docs from 1000 to 1960 2021-05-17 09:09:55,282 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 4 docs from 0 to 1000 2021-05-17 09:09:55,285 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 17 docs from 0 to 1000 2021-05-17 09:09:55,285 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 09:09:55,286 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 15 docs from 1000 to 1960 2021-05-17 09:09:55,286 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 09:09:55,288 | test | ERROR | pool-24-thread-24 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-17 09:17:06,400 | test | INFO | pool-24-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******X | 2.000 | | 1.766 | ****X | 1.766 | | 0.500 | ****X | 0.500 | | 0.000 | *********.....****.********......*****.*...*****...****X | 64.241 | | 59.743 | .................X | 59.743 | | 53.373 | ...............******..****X | 61.366 | | 52.862 | ...............*****.X | 57.556 | | 53.970 | ...............*****...........................................................................****X | 296.931 | | 259.708 | ..............................................................................X | 259.708 | | 123.712 | .....................................****X | 123.712 | | 120.514 | ....................................******X | 123.043 | | 108.371 | ................................****X | 108.371 | | 97.430 | ............................******.*****..............................*.....****X | 213.427 | | 118.773 | ...................................****X | 118.773 | | 110.734 | .................................X | 110.734 | | 109.249 | ................................*****......****X | 127.063 | | 123.540 | .....................................X | 123.540 | | 114.750 | ..................................*****.****X | 118.456 | | 115.354 | ..................................*****X | 115.559 | | 114.457 | ..................................*.*****.............*****X | 162.060 | | 102.277 | ..............................*****X | 102.277 | | 60.594 | .................***********..............******....................................................................*****X | 325.091 | | 55.501 | ................****X | 55.501 | | 11.600 | ..X | 11.600 | | 0.999 | **********X | 3.198 | | 2.000 | ****X | 2.000 | | 1.599 | ******............................****X | 93.778 | | 93.380 | ...........................*..*****.....*****X | 115.388 | | 111.571 | .................................*****X | 113.169 | | 106.886 | ...............................*****...******..........................****X | 200.021 | | 144.768 | ...........................................X | 144.768 | | 116.603 | ..................................*****.....*****X | 132.779 | | 92.390 | ...........................****X | 92.390 | | 81.585 | ........................X | 81.585 | | 81.487 | ........................****X | 81.487 | | 42.989 | ............*****X | 42.989 | | 2.300 | ******..................****X | 61.301 | | 55.406 | ................X | 55.406 | | 36.794 | ..........****X | 36.794 | | 2.599 | ****X | 2.599 | | 1.400 | X | 1.400 | | 1.200 | *****......******..........................*.......****X | 129.161 | | 81.408 | ........................****X | 81.408 | | 62.401 | ..................X | 62.401 | | 48.593 | .............****X | 48.593 | | 37.196 | ..........X | 37.196 | | 1.600 | ****X | 1.600 | | 0.800 | ***********.*....******.................................*****..*.****X | 135.733 | | 28.197 | .......X | 28.197 | | 3.000 | ***X | 3.000 | | 2.600 | X | 2.600 | | 2.200 | ****X | 2.200 | | 2.000 | **********X | 2.600 | | 2.000 | X | 2.000 | | 0.800 | *****..............*..........*****.......*....***X | 114.203 | | 109.803 | ................................X | 109.803 | | 83.232 | ........................****X | 83.232 | | 66.530 | ...................X | 66.530 | | 2.500 | *****.............*....****X | 56.364 | | 46.770 | .............*****X | 46.770 | | 1.200 | *****...............*.........*****.****X | 83.026 | | 19.479 | ....X | 19.479 | | 2.201 | ****X | 2.201 | | 1.202 | *****.......*............****X | 62.422 | | 23.209 | ......*****X | 23.209 | +---------+----------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:17:07,332 | test | INFO | pool-24-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***................*..................**X | 107.800 | | 86.005 | ..........................X | 86.005 | | 26.674 | .......****...............*.............................................***X | 213.652 | | 182.496 | .........................................................X | 182.496 | | 103.914 | ................................****.....*....**X | 131.777 | | 99.888 | ..............................X | 99.888 | | 88.545 | ...........................**X | 88.545 | | 37.864 | ...........****X | 37.864 | | 7.202 | .****......................*.......***...*.....*****.....................................*****..........*..............****X | 313.034 | | 167.198 | ....................................................X | 167.198 | | 135.315 | ..........................................****X | 135.315 | | 132.809 | .........................................X | 132.809 | | 112.837 | ...................................****X | 112.837 | | 103.635 | ................................*..****..X | 117.720 | | 114.738 | ...................................****X | 114.738 | | 109.636 | ..................................X | 109.636 | | 85.546 | ..........................**********X | 85.835 | | 64.885 | ...................****X | 64.885 | | 36.874 | ..........******.........*..****.*....****X | 86.531 | | 79.587 | ........................*****..*..*****......*****X | 112.165 | | 54.278 | ................**********X | 55.577 | | 46.385 | .............******..*****..X | 58.788 | | 57.987 | .................****X | 57.987 | | 51.788 | ...............X | 51.788 | | 32.792 | .........*****............******....****X | 81.979 | | 72.678 | ......................X | 72.678 | | 41.689 | ............****X | 41.689 | | 19.094 | .....******.............*****...*..*****.......*..........................****X | 180.171 | | 82.789 | .........................*****X | 82.789 | | 77.282 | .......................****X | 77.282 | | 49.286 | ..............******........*****.....*..*****..........*****X | 125.768 | | 62.592 | ..................*****X | 62.592 | | 53.092 | ...............*****..******.****X | 60.501 | | 58.691 | .................****X | 58.691 | | 45.695 | .............******..............*****...*..**********X | 105.674 | | 22.900 | ......*****X | 22.900 | | 2.000 | *******X | 2.200 | | 2.200 | ***X | 2.200 | | 1.000 | **********.******.................*****X | 58.506 | | 41.393 | ............****X | 41.393 | | 17.397 | ....X | 17.397 | | 1.800 | *****............................*****X | 88.695 | | 73.495 | ......................X | 73.495 | | 15.305 | ...*****..............*****X | 58.692 | | 42.005 | ............****X | 42.005 | | 8.840 | .X | 8.840 | | 1.800 | ****X | 1.800 | | 1.100 | X | 1.100 | | 1.000 | ***X | 1.000 | | 0.300 | X | 0.300 | | 0.200 | *****.*........*****.....*...****X | 55.852 | | 0.200 | **********X | 1.400 | | 1.400 | X | 1.400 | | 1.200 | *****...............*....****X | 60.053 | | 22.285 | ......****X | 22.285 | | 9.586 | ..X | 9.586 | | 1.400 | *****....*....*****.........*****X | 53.663 | | 35.888 | ..........*****...*.****X | 48.311 | | 12.346 | ..X | 12.346 | | 2.502 | ****X | 2.502 | | 1.002 | X | 1.002 | | 0.701 | *********X | 2.401 | | 1.600 | *****X | 1.600 | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:17:08,753 | test | INFO | pool-24-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***........................................................................................***X | 625.505 | | 269.327 | ....................................*****.........................................................***X | 673.662 | | 443.582 | .............................................................X | 443.582 | | 240.233 | ................................****...........*...........................***.......***X | 555.473 | | 140.369 | ..................****.............*......**X | 270.215 | | 211.932 | ............................X | 211.932 | | 185.139 | .........................***X | 189.337 | | 166.399 | ......................*****.........................****X | 346.431 | | 263.885 | ....................................X | 263.885 | | 236.680 | ................................*****................******...............................****X | 573.333 | | 361.281 | .................................................*****X | 361.281 | | 309.933 | ..........................................****X | 309.933 | | 287.981 | .......................................X | 287.981 | | 242.532 | .................................*****.....*......................***X | 436.269 | | 436.269 | ............................................................X | 436.269 | | 242.239 | .................................****X | 242.239 | | 175.176 | .......................***X | 175.176 | | 175.176 | .......................**.................................****X | 409.338 | | 401.256 | .......................................................X | 401.256 | | 384.109 | .....................................................****X | 384.109 | | 231.148 | ...............................******...*****..****X | 264.912 | | 202.814 | ...........................******.............*****X | 291.293 | | 239.418 | ................................****X | 239.418 | | 135.463 | ..................******....................*****....*......***X | 349.420 | | 349.420 | ................................................*....*..........****X | 449.464 | | 265.843 | ....................................******........****X | 321.320 | | 319.320 | ...........................................*.****X | 321.999 | | 296.415 | ........................................X | 296.415 | | 259.196 | ...................................*****..*................*****.*****X | 396.942 | | 357.557 | .................................................*X | 357.557 | | 357.557 | .................................................***...............................****X | 578.345 | | 385.966 | .....................................................X | 385.966 | | 284.035 | .......................................*****.....*****X | 321.596 | | 267.674 | ....................................**X | 267.674 | | 267.674 | ....................................*X | 267.674 | | 265.640 | ....................................**X | 265.640 | | 265.640 | ....................................***.................................****X | 502.038 | | 502.038 | .....................................................................X | 502.038 | | 365.345 | ..................................................****X | 365.345 | | 317.065 | ...........................................****X | 317.065 | | 256.748 | ...................................X | 256.748 | | 233.305 | ...............................****X | 233.305 | | 201.273 | ...........................****X | 201.273 | | 197.974 | ..........................******...*****.....X | 255.135 | | 253.035 | ..................................*****............*****X | 334.515 | | 289.305 | .......................................*****...*.........*****.........................*X | 552.289 | | 552.289 | ............................................................................**X | 552.289 | | 433.609 | ............................................................*****X | 433.609 | | 406.648 | ........................................................****X | 406.648 | | 291.034 | .......................................X | 291.034 | | 243.129 | .................................****X | 243.129 | | 220.623 | ..............................******.....******....................*****X | 398.765 | | 217.878 | .............................*****................*............****X | 413.645 | | 306.453 | ..........................................*****X | 306.453 | | 205.569 | ...........................X | 205.569 | | 178.242 | ........................*****................*****............*........****X | 437.733 | | 413.338 | .........................................................X | 413.338 | | 402.638 | .......................................................****X | 402.638 | | 386.114 | .....................................................*****.X | 392.392 | | 347.576 | ...............................................*****.........................*.........****X | 585.731 | | 444.169 | .............................................................X | 444.169 | | 329.458 | .............................................***X | 329.458 | | 310.843 | ..........................................X | 310.843 | | 298.743 | .........................................****X | 298.743 | | 293.357 | ........................................X | 293.357 | | 181.349 | ........................*****.......................................****X | 457.309 | | 444.281 | .............................................................*..****X | 456.559 | | 407.047 | ........................................................X | 407.047 | | 336.598 | ..............................................*****..****X | 354.311 | | 335.316 | ..............................................*..***X | 349.881 | | 311.669 | ..........................................X | 311.669 | | 249.814 | ..................................*****.........................*.........*****.............****X | 582.377 | | 464.857 | ................................................................******...........*****......*..................X | 710.036 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:17:08,828 | test | WARNING | Thread-77585 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:RZ2sSWVCxUEaYB-42-703000:I1%svD1B-42-840000 create failed keys from task: [u'test_collections-2450'] 2021-05-17 09:17:08,828 | test | WARNING | Thread-77586 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:RZ2sSWVCxUEaYB-42-703000:P-42-839000 create failed keys from task: [u'test_collections-2450'] 2021-05-17 09:17:08,841 | test | WARNING | Thread-77625 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:g-42-122000:u6lXOUhLtsRVgu3YgiJ-42-124000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 09:17:08,875 | test | WARNING | Thread-77764 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 update failed keys from task: [u'test_collections-1807', u'test_collections-1917', u'test_collections-1887', u'test_collections-1944', u'test_collections-1923', u'test_collections-1901', u'test_collections-1935', u'test_collections-1957', u'test_collections-1891', u'test_collections-1761'] 2021-05-17 09:17:08,877 | test | WARNING | Thread-77767 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:-SUuiHHJz-1-814000:pSgP_LfU%fUStgHUWC-1-943000 update failed keys from task: [u'test_collections-1190', u'test_collections-1028', u'test_collections-1149', u'test_collections-1106', u'test_collections-1109', u'test_collections-1186', u'test_collections-1000', u'test_collections-1022', u'test_collections-1045', u'test_collections-1016', u'test_collections-1138', u'test_collections-1096', u'test_collections-1099', u'test_collections-1132', u'test_collections-1177'] 2021-05-17 09:17:08,877 | test | WARNING | Thread-77768 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:n1Iz4LJLjZWb6-46-129000:s-46-678000 create failed keys from task: [u'test_collections-830'] 2021-05-17 09:17:08,904 | test | WARNING | Thread-77847 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:V53XUL3Zts4gigdmiDD-2-138000 update failed keys from task: [u'test_collections-1444', u'test_collections-1401', u'test_collections-1325', u'test_collections-1547', u'test_collections-1520', u'test_collections-1542', u'test_collections-1576', u'test_collections-1554', u'test_collections-1533', u'test_collections-1458', u'test_collections-1539', u'test_collections-1591', u'test_collections-1475'] 2021-05-17 09:17:08,907 | test | WARNING | Thread-77854 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:UCG-42-149000 update failed keys from task: [u'test_collections-1412', u'test_collections-1401'] 2021-05-17 09:17:08,910 | test | WARNING | Thread-77770 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9KnnReY1QD-47-523000:ly0DdlZAfv5_Pc-47-620000 create failed keys from task: [u'test_collections-812', u'test_collections-982'] 2021-05-17 09:17:08,911 | test | WARNING | Thread-77857 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:FsX_xjZauPHZAYE%-42-149000 update failed keys from task: [u'test_collections-1059', u'test_collections-982', u'test_collections-1016'] 2021-05-17 09:17:08,910 | test | WARNING | Thread-77733 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7-42-140000 update failed keys from task: [u'test_collections-1412', u'test_collections-1401'] 2021-05-17 09:17:08,911 | test | WARNING | Thread-77849 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:d-42-148000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 09:17:09,013 | test | WARNING | Thread-77857 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:FsX_xjZauPHZAYE%-42-149000 delete failed keys from task: [u'test_collections-517', u'test_collections-606', u'test_collections-615', u'test_collections-621', u'test_collections-566', u'test_collections-632', u'test_collections-523', u'test_collections-677', u'test_collections-664', u'test_collections-650'] 2021-05-17 09:17:09,247 | test | WARNING | Thread-77847 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pd7XdB%AI-42-147000:V53XUL3Zts4gigdmiDD-2-138000 delete failed keys from task: [u'test_collections-317', u'test_collections-304', u'test_collections-287', u'test_collections-294'] 2021-05-17 09:17:09,247 | test | WARNING | Thread-77764 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Y%sR-1-433000:qk-4XfppVnKm5Y-1-554000 delete failed keys from task: [u'test_collections-208', u'test_collections-304', u'test_collections-326', u'test_collections-247', u'test_collections-369', u'test_collections-202', u'test_collections-341', u'test_collections-363', u'test_collections-67', u'test_collections-236', u'test_collections-357', u'test_collections-279', u'test_collections-398', u'test_collections-294', u'test_collections-273', u'test_collections-392', u'test_collections-370'] 2021-05-17 09:17:22,075 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 09:18:40,430 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 09:18:40,430 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 09:18:57,266 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 09:18:57,266 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 09:19:19,221 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 09:19:19,223 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 09:20:12,980 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.132999897 seconds to complete 2021-05-17 09:20:12,996 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3352.455s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 22 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_22 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_23 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_23', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 23, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 09:20:22,436 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #23 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 09:20:23,118 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #23 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 09:20:23,573 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd1218c type=client open_count=1 channel=[id: 0x624eb1ba, 0.0.0.0/0.0.0.0:43614] timeout=300.0>) 2021-05-17 09:20:44,390 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd1218d type=client open_count=1 channel=[id: 0x9f85f1b3, 0.0.0.0/0.0.0.0:46608] timeout=300.0>) 2021-05-17 09:20:46,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd1218e type=client open_count=1 channel=[id: 0x31adf447, 0.0.0.0/0.0.0.0:33862] timeout=300.0>) 2021-05-17 09:20:48,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xd1218f type=client open_count=1 channel=[id: 0x5f63568d, 0.0.0.0/0.0.0.0:60975] timeout=300.0>) 2021-05-17 09:21:12,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xd12190 type=client open_count=1 channel=[id: 0xf1240df8, 0.0.0.0/0.0.0.0:38491] timeout=300.0>) 2021-05-17 09:21:14,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xd12191 type=client open_count=1 channel=[id: 0xbf20f144, 0.0.0.0/0.0.0.0:37381] timeout=300.0>) 2021-05-17 09:21:16,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xd12192 type=client open_count=1 channel=[id: 0xf1937ce1, 0.0.0.0/0.0.0.0:48152] timeout=300.0>) 2021-05-17 09:21:40,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xd12193 type=client open_count=1 channel=[id: 0x1036de06, 0.0.0.0/0.0.0.0:49889] timeout=300.0>) 2021-05-17 09:21:42,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xd12194 type=client open_count=1 channel=[id: 0x35910342, 0.0.0.0/0.0.0.0:35939] timeout=300.0>) 2021-05-17 09:21:44,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xd12195 type=client open_count=1 channel=[id: 0x5b3ad98a, 0.0.0.0/0.0.0.0:60843] timeout=300.0>) 2021-05-17 09:22:08,457 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xd12196 type=client open_count=1 channel=[id: 0x8bcbb595, 0.0.0.0/0.0.0.0:42051] timeout=300.0>) 2021-05-17 09:22:10,474 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xd12197 type=client open_count=1 channel=[id: 0x401abb21, 0.0.0.0/0.0.0.0:43335] timeout=300.0>) 2021-05-17 09:22:12,492 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xd12198 type=client open_count=1 channel=[id: 0x7bbff796, 0.0.0.0/0.0.0.0:53397] timeout=300.0>) 2021-05-17 09:22:36,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xd12199 type=client open_count=1 channel=[id: 0x20c0489f, 0.0.0.0/0.0.0.0:42150] timeout=300.0>) 2021-05-17 09:22:38,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xd1219a type=client open_count=1 channel=[id: 0x9014a402, 0.0.0.0/0.0.0.0:52026] timeout=300.0>) 2021-05-17 09:22:40,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 09:22:57,615 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 09:23:11,489 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 09:23:11,510 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #23 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 09:23:11,548 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #23 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 09:23:24,569 | test | INFO | pool-25-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.209993000233 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 09:23:24,601 | test | INFO | pool-25-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:23:34,686 | test | INFO | pool-25-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 09:23:34,713 | test | INFO | pool-25-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1440000534 sec 2021-05-17 09:23:34,983 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 09:24:30,260 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 09:24:30,552 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 606188040 | 99898521 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 574842576 | 133423448 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 655409624 | 78872232 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 09:24:30,552 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 09:25:14,130 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 09:25:14,145 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 09:41:58,076 | test | INFO | pool-25-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *..........*****..............................*....................................................*........***X | 42.003 | | 10.596 | ........................X | 10.596 | | 0.000 | **********************************...............................*****...........*****X | 18.002 | | 6.101 | .............*****.....................*......................*...***X | 25.501 | | 21.301 | .................................................X | 21.301 | | 4.499 | .........**********...........*...****X | 10.499 | | 8.399 | ..................X | 8.399 | | 0.000 | ***************.*......***X | 3.000 | | 2.300 | ....X | 2.300 | | 1.500 | ..*****.*..*****.....****X | 4.900 | | 3.700 | .......X | 3.700 | | 1.700 | ...X | 1.700 | | 1.500 | ..****...*.****X | 3.000 | | 2.500 | ....******..............****X | 8.000 | | 6.400 | ..............X | 6.400 | | 3.300 | ......X | 3.300 | | 3.000 | ......***X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | *****.......................*.*****.......*****X | 13.101 | | 0.400 | *****..*****X | 1.000 | | 1.000 | .*..****X | 2.000 | | 1.101 | .****X | 1.101 | | 0.500 | X | 0.500 | | 0.400 | ****X | 0.400 | | 0.000 | **********************...*...*.***X | 2.999 | | 1.600 | ..X | 1.600 | | 1.500 | ..*****..*.....****X | 4.500 | | 1.500 | ..****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | ***********....................................*...***X | 16.501 | | 9.300 | .....................X | 9.300 | | 8.899 | ....................****X | 8.899 | | 7.601 | .................****X | 7.601 | | 0.400 | X | 0.400 | | 0.000 | ****************...*****X | 1.500 | | 0.000 | ************...****X | 1.500 | | 0.000 | ******...****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | *****..*.**********X | 1.554 | | 0.000 | *****................*.****X | 7.500 | | 0.000 | ******.....................******..............****X | 14.999 | | 1.000 | .X | 1.000 | | 0.000 | *****...*****.....*..****X | 4.500 | | 0.000 | ****************************...****X | 1.500 | | 0.000 | *****************...*****X | 1.499 | | 0.000 | *****************..*.***X | 1.499 | | 0.999 | .X | 0.999 | | 0.500 | *****..*.............****X | 6.993 | | 4.496 | .........****X | 4.496 | | 1.499 | ..*****X | 1.499 | | 0.499 | X | 0.499 | | 0.000 | **********.......**********X | 2.998 | | 0.200 | X | 0.200 | | 0.000 | ********************************************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 09:41:58,496 | test | INFO | pool-25-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......*****......................................................................*.......................****X | 101.407 | | 7.291 | ......X | 7.291 | | 0.000 | *********************************.....****X | 5.999 | | 5.901 | ....X | 5.901 | | 5.901 | ....*****......*****X | 12.101 | | 12.000 | ..........****X | 12.000 | | 3.600 | ..X | 3.600 | | 3.000 | .*****..*.......****X | 12.001 | | 4.499 | ...*****X | 4.499 | | 0.000 | ****************..*...........***X | 13.499 | | 9.899 | ........X | 9.899 | | 0.900 | X | 0.900 | | 0.000 | ****.******.*.....****X | 7.500 | | 0.000 | ***********...*.****X | 4.500 | | 3.600 | ..X | 3.600 | | 0.000 | *****.*****X | 1.500 | | 0.000 | *****..*...**********X | 6.000 | | 3.000 | .*****X | 3.000 | | 0.000 | ******..*****...****X | 6.002 | | 4.599 | ...X | 4.599 | | 4.499 | ...****X | 4.499 | | 3.599 | ..X | 3.599 | | 0.000 | ***********************....****X | 4.499 | | 2.301 | .*****X | 2.701 | | 2.100 | .****X | 2.200 | | 1.500 | ****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | ********************************************.******.*****X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | ****..*..****X | 4.500 | | 3.900 | ..X | 3.900 | | 1.100 | **********X | 1.900 | | 0.000 | ************.*****.******.......*****X | 10.100 | | 1.900 | *****...*****X | 4.499 | | 1.500 | X | 1.500 | | 0.000 | ***************************.****X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | ****X | 1.500 | | 0.000 | *****......*.***X | 7.498 | | 6.298 | .....X | 6.298 | | 1.500 | ****X | 1.500 | | 1.498 | X | 1.498 | | 1.498 | ***X | 1.498 | | 0.100 | X | 0.100 | | 0.000 | ***********..****X | 2.997 | | 0.200 | X | 0.200 | | 0.000 | **************************************..******.*.***X | 4.496 | | 3.596 | ..X | 3.596 | | 0.000 | *****....*****X | 4.097 | | 1.898 | *****X | 1.898 | | 0.000 | ******.****X | 1.499 | | 0.000 | *****************..*****X | 2.998 | | 0.000 | ***********************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:41:59,262 | test | INFO | pool-25-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.*****........................................................*......................................****X | 764.394 | | 762.994 | .............................................................................................*.****X | 770.594 | | 676.137 | ...................................................................................****X | 676.137 | | 662.003 | .................................................................................X | 662.003 | | 628.189 | .............................................................................*****........*...*****..........*****X | 803.405 | | 704.314 | ......................................................................................****X | 704.314 | | 570.987 | ......................................................................X | 570.987 | | 463.247 | ........................................................*****.X | 468.761 | | 415.942 | ..................................................****X | 415.942 | | 377.913 | ..............................................****X | 377.913 | | 348.594 | ..........................................X | 348.594 | | 343.710 | .........................................*****...*.................*********X | 501.459 | | 491.443 | ............................................................X | 491.443 | | 480.943 | ..........................................................*****...*****.*...*...***X | 557.431 | | 538.402 | ..................................................................X | 538.402 | | 467.011 | .........................................................****..*.****X | 496.442 | | 484.292 | ...........................................................******....****X | 519.829 | | 514.909 | ...............................................................X | 514.909 | | 465.477 | ........................................................X | 465.477 | | 449.860 | ......................................................****....*....****X | 509.736 | | 486.491 | ...........................................................X | 486.491 | | 448.037 | ......................................................****X | 448.037 | | 427.256 | ....................................................*****.*.X | 446.565 | | 440.376 | .....................................................****.......*..*****...*****X | 536.735 | | 458.091 | ........................................................*****X | 464.118 | | 456.606 | .......................................................X | 456.606 | | 453.407 | .......................................................****...X | 479.644 | | 476.363 | ..........................................................*****.*****.X | 492.571 | | 484.090 | ...........................................................X | 484.090 | | 481.480 | ..........................................................***X | 481.480 | | 459.037 | ........................................................X | 459.037 | | 446.627 | ......................................................*****.......******.....****X | 540.263 | | 520.509 | ...............................................................X | 520.509 | | 509.109 | ..............................................................****X | 509.109 | | 502.999 | .............................................................*...****X | 522.488 | | 492.540 | ............................................................****X | 492.540 | | 485.568 | ...........................................................*.****X | 491.858 | | 459.688 | ........................................................*****X | 459.688 | | 446.599 | ......................................................*****........*****..*.*****X | 534.198 | | 481.602 | ..........................................................****X | 481.602 | | 431.043 | ....................................................****X | 431.043 | | 375.915 | .............................................X | 375.915 | | 301.178 | ....................................*****....*.......................****X | 517.957 | | 506.713 | ..............................................................******....X | 545.243 | | 528.830 | ................................................................***X | 528.830 | | 483.847 | ...........................................................X | 483.847 | | 475.637 | ..........................................................****X | 475.637 | | 455.155 | .......................................................*.****X | 460.426 | | 455.221 | .......................................................******.........****X | 525.088 | | 439.015 | .....................................................******.*****....*.....*****.........*...****X | 614.686 | | 553.893 | ...................................................................X | 553.893 | | 499.208 | .............................................................*****...X | 525.263 | | 522.373 | ................................................................***X | 522.373 | | 516.541 | ...............................................................X | 516.541 | | 497.420 | ............................................................****X | 497.420 | | 479.345 | ..........................................................X | 479.345 | | 389.903 | ...............................................*****.........*****...*.****X | 492.742 | | 488.942 | ...........................................................X | 488.942 | | 482.044 | ...........................................................*****.****X | 497.532 | | 447.735 | ......................................................X | 447.735 | | 434.329 | .....................................................*****................****X | 563.199 | | 558.565 | ....................................................................******.X | 565.699 | | 524.639 | ................................................................****X | 524.639 | | 454.507 | .......................................................******.*****....****X | 496.127 | | 491.255 | ............................................................X | 491.255 | | 463.877 | ........................................................*****....****X | 496.494 | | 485.813 | ...........................................................X | 485.813 | | 451.965 | .......................................................*****.******..*****....******.X | 515.114 | | 490.107 | ............................................................****..*....****X | 538.618 | | 483.995 | ...........................................................******....X | 517.826 | | 513.336 | ..............................................................****...*.****X | 541.526 | | 528.057 | ................................................................******..****X | 545.516 | | 540.616 | ..................................................................X | 540.616 | | 524.485 | ................................................................****X | 524.485 | | 445.188 | ......................................................***********...*..**********X | 488.252 | | 469.146 | .........................................................X | 469.146 | | 431.894 | ....................................................****..*...****X | 468.734 | | 369.434 | ............................................X | 369.434 | | 350.237 | ..........................................*****...................****X | 500.357 | | 499.627 | .............................................................*.****X | 511.098 | | 460.647 | ........................................................X | 460.647 | | 418.273 | ...................................................*****.......*...****X | 505.784 | | 464.903 | ........................................................X | 464.903 | | 462.382 | ........................................................*****.............****X | 566.735 | | 485.143 | ...........................................................X | 485.143 | | 474.843 | ..........................................................X | 474.843 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:41:59,519 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 60.7289878502 | 6437847040 | 4363546624 | 2383495168 / 7511588864 | 101047 / 204198 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 80.7587985689 | 6437847040 | 4247134208 | 2452926464 / 7511588864 | 99207 / 201652 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 66.1183333333 | 6437847040 | 4285775872 | 2397134848 / 7511588864 | 99465 / 202202 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 80.4669922168 | 6437847040 | 3855450112 | 3043569664 / 9530753024 | 101711 / 197186 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 70.4671588807 | 6437847040 | 4261310464 | 2465775616 / 7511588864 | 102052 / 201736 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 09:42:13,131 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 09:43:49,878 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164002664 | 260840123 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 161805504 | 285463642 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 443985224 | 461870625 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 09:43:49,880 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #23 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 09:44:04,055 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 09:44:04,055 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 09:44:04,056 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 09:44:14,874 | test | INFO | pool-25-thread-3 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.20000004768 seconds to complete 2021-05-17 09:44:14,880 | test | INFO | pool-25-thread-3 | [common_lib:sleep:22] Sleep 8.20000004768 seconds. Reason: Wait after rebalance complete 2021-05-17 09:46:23,154 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0720000267029 seconds 2021-05-17 09:46:23,154 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 09:46:23,155 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 09:46:30,072 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 09:57:34,964 | test | INFO | pool-25-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***......******....................******................................*.....................***X | 228.154 | | 228.154 | ..............................................................................*..*****X | 232.140 | | 180.501 | ..............................................................******.......*****.............*....*****.....*****X | 263.243 | | 217.211 | ..........................................................................****X | 217.211 | | 215.028 | ..........................................................................*...............*****....*......****X | 285.872 | | 187.530 | ................................................................*****..*.*****...*....*****.....****X | 228.980 | | 160.225 | .......................................................X | 160.225 | | 113.808 | ......................................****X | 113.808 | | 109.085 | .....................................*****X | 109.085 | | 92.084 | ...............................*****X | 92.084 | | 89.376 | ..............................*****X | 89.376 | | 80.084 | ...........................*****X | 80.084 | | 28.001 | ........*X | 28.001 | | 28.001 | ........***X | 28.001 | | 13.401 | ...X | 13.401 | | 1.300 | ****X | 1.300 | | 0.601 | ****************X | 2.201 | | 1.900 | X | 1.900 | | 1.700 | *****X | 2.001 | | 1.901 | *X | 1.901 | | 1.901 | *********.******X | 4.201 | | 3.102 | ****X | 3.102 | | 2.701 | X | 2.701 | | 2.400 | ****X | 2.400 | | 2.400 | *X | 2.800 | | 2.499 | ***X | 2.499 | | 1.999 | X | 1.999 | | 1.500 | X | 1.500 | | 1.200 | ***X | 1.200 | | 0.700 | *X | 0.700 | | 0.600 | **********X | 2.099 | | 1.300 | **********X | 2.000 | | 1.499 | ******X | 2.400 | | 2.400 | ****X | 2.400 | | 1.000 | ************X | 2.500 | | 2.100 | **********X | 2.800 | | 1.999 | *****X | 1.999 | | 1.500 | ************X | 2.000 | | 1.800 | *****X | 1.800 | | 1.000 | *****X | 1.100 | | 1.100 | **********X | 2.499 | | 1.600 | *****X | 1.600 | | 1.499 | ***********X | 2.099 | | 1.200 | ******.****X | 2.999 | | 1.200 | ***********************X | 2.799 | | 0.400 | X | 0.400 | | 0.000 | *****************X | 2.200 | | 2.000 | ****X | 2.000 | | 1.699 | X | 1.699 | | 1.600 | *****X | 1.600 | | 0.700 | **********X | 0.900 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:57:35,993 | test | INFO | pool-25-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....******..................******..............................................*****.......*****X | 199.437 | | 197.993 | ..........................................................................******....*****X | 207.997 | | 207.932 | ..............................................................................******................*****...*..*****X | 261.986 | | 229.489 | ......................................................................................*****..X | 234.479 | | 223.018 | ....................................................................................X | 223.018 | | 223.018 | ....................................................................................***X | 223.018 | | 218.662 | ..................................................................................*****.......X | 236.228 | | 216.350 | .................................................................................*****.....*****X | 229.725 | | 161.855 | ............................................................****X | 161.855 | | 128.505 | ................................................X | 128.505 | | 109.628 | ........................................****X | 109.628 | | 72.087 | ..........................*****X | 72.087 | | 36.975 | .............X | 36.975 | | 2.801 | ******.***X | 5.303 | | 3.004 | X | 3.004 | | 0.700 | **********X | 1.800 | | 1.101 | X | 1.101 | | 0.700 | ****X | 0.700 | | 0.700 | ***********X | 2.201 | | 1.400 | ********X | 1.701 | | 1.701 | **X | 1.701 | | 1.001 | *************.**********X | 4.300 | | 2.400 | *****X | 2.400 | | 1.200 | ***********X | 1.200 | | 0.200 | ***********X | 2.000 | | 1.100 | ******.*****X | 2.900 | | 1.699 | *****X | 1.699 | | 0.500 | *****X | 0.500 | | 0.300 | *****************X | 2.500 | | 0.600 | ***********X | 1.499 | | 0.900 | ******.***********X | 3.800 | | 2.400 | X | 2.400 | | 1.300 | ****X | 1.300 | | 1.100 | ******.****X | 2.899 | | 2.299 | X | 2.299 | | 1.400 | *****X | 1.400 | | 0.600 | ***********X | 0.900 | | 0.900 | ******.*****X | 4.099 | | 1.400 | ****X | 1.400 | | 1.200 | ******X | 1.200 | | 0.400 | *****.*****X | 5.198 | | 0.200 | ******.*****X | 2.900 | | 2.599 | *****X | 2.599 | | 1.900 | ******X | 1.900 | | 0.700 | **********X | 1.600 | | 1.000 | *****X | 1.000 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 09:57:36,963 | test | INFO | pool-25-thread-28 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***..****X | 48.047 | | 23.355 | X | 23.355 | | 0.000 | ******....................................................................................................****X | 1606.159 | | 795.472 | ................................................*****X | 795.472 | | 678.435 | .........................................*X | 678.435 | | 678.435 | .........................................***X | 678.435 | | 655.856 | .......................................****X | 655.856 | | 577.873 | ..................................X | 577.873 | | 543.422 | ................................*****.........*****X | 685.215 | | 550.864 | .................................X | 550.864 | | 389.492 | .......................*****...******.....*****X | 524.842 | | 356.718 | .....................****X | 356.718 | | 318.988 | ..................X | 318.988 | | 198.355 | ...........*****..*......****X | 327.754 | | 212.219 | ............*****....*.....*****.........******........*........*****.*.***X | 792.543 | | 621.117 | .....................................X | 621.117 | | 470.295 | ............................*****....*.**X | 549.531 | | 549.531 | .................................***X | 552.632 | | 552.632 | .................................*X | 552.632 | | 552.632 | .................................**...................****X | 859.904 | | 819.372 | ..................................................X | 819.372 | | 722.585 | ...........................................****X | 722.585 | | 680.354 | .........................................X | 680.354 | | 573.250 | ..................................*****...*.......*X | 730.057 | | 730.057 | ............................................**X | 730.057 | | 722.065 | ...........................................*..****X | 745.476 | | 647.079 | .......................................X | 647.079 | | 617.047 | .....................................*****....*.****X | 700.933 | | 631.295 | ......................................X | 631.295 | | 613.766 | .....................................****X | 613.766 | | 542.471 | ................................X | 542.471 | | 496.286 | .............................*****..*.**********X | 539.598 | | 506.088 | ..............................X | 506.088 | | 476.095 | ............................**X | 476.095 | | 476.095 | ............................*X | 476.095 | | 453.472 | ...........................******...*****X | 501.558 | | 496.482 | .............................*X | 496.482 | | 496.482 | .............................***X | 496.482 | | 489.451 | .............................*.*****.X | 523.684 | | 451.860 | ...........................***X | 451.860 | | 451.860 | ...........................X | 451.860 | | 436.348 | ..........................******.......X | 548.030 | | 548.030 | .................................**X | 548.030 | | 548.030 | .................................********.*****X | 564.402 | | 512.638 | ..............................*****X | 512.638 | | 503.769 | ..............................*.****X | 525.465 | | 449.631 | ..........................X | 449.631 | | 423.196 | .........................*****..*.*****...*..****X | 555.324 | | 508.981 | ..............................*****X | 508.981 | | 414.163 | ........................*****.....*****X | 487.241 | | 452.026 | ...........................******......*****X | 550.584 | | 531.853 | ................................X | 531.853 | | 522.229 | ...............................*****...****X | 577.920 | | 568.925 | ..................................X | 568.925 | | 563.289 | ..................................****X | 563.289 | | 469.366 | ............................******.........*****X | 621.515 | | 580.781 | ...................................*****X | 580.781 | | 568.129 | ..................................*****X | 568.129 | | 550.884 | .................................******....*****...*..****X | 691.777 | | 634.357 | ......................................X | 634.357 | | 617.440 | .....................................*****.******.........*****X | 778.399 | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 09:57:37,209 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 09:57:37,259 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 09:57:47,707 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 09:57:47,894 | test | INFO | pool-25-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.288328527858 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 45.4156041135 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 27.0845485909 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 48.6458107635 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 42.9664322144 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 09:57:47,944 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:57:53,052 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:57:53,436 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 09:57:58,115 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:58:03,180 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 09:58:08,259 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.47 2021-05-17 09:58:13,332 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.84 2021-05-17 09:58:18,427 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 11.24 2021-05-17 09:58:23,522 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 16.6 2021-05-17 09:58:28,668 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 22.04 2021-05-17 09:58:33,799 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 27.58 2021-05-17 09:58:38,927 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 32.78 2021-05-17 09:58:44,032 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 34.39 2021-05-17 09:58:49,141 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 38.63 2021-05-17 09:58:52,319 | test | INFO | pool-25-thread-24 | [table_view:display:72] Keys failed in bucket2:JJPZ-51-166000:3_-51-316000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,319 | test | INFO | pool-25-thread-7 | [table_view:display:72] Keys failed in bucket2:v-RTOyeJDWbUFs8Az-51-469000:QjsFGKwF-51-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2014 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,319 | test | INFO | pool-25-thread-9 | [table_view:display:72] Keys failed in bucket2:JJPZ-51-166000:nbTaH-MIdH-jMTd5TuM-51-317000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2014 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,319 | test | INFO | pool-25-thread-24 | [table_view:display:72] Keys failed in bucket2:JJPZ-51-166000:3_-51-316000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,319 | test | INFO | pool-25-thread-7 | [table_view:display:72] Keys failed in bucket2:v-RTOyeJDWbUFs8Az-51-469000:QjsFGKwF-51-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2014 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,321 | test | INFO | pool-25-thread-9 | [table_view:display:72] Keys failed in bucket2:JJPZ-51-166000:nbTaH-MIdH-jMTd5TuM-51-317000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2014 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,322 | test | INFO | pool-25-thread-29 | [table_view:display:72] Keys failed in bucket2:tSRhw-wd2s-50-866000:yITBA-51-17000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2014 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:52,322 | test | INFO | pool-25-thread-29 | [table_view:display:72] Keys failed in bucket2:tSRhw-wd2s-50-866000:yITBA-51-17000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2184 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2014 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:54,226 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 41.53 2021-05-17 09:58:55,269 | test | INFO | pool-25-thread-19 | [table_view:display:72] Keys failed in bucket2:xV1RBonbPrQiAa-26-782000:1UlbM-26-889000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:58:59,351 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 44.72 2021-05-17 09:59:04,437 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 46.6 2021-05-17 09:59:09,520 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.83 2021-05-17 09:59:11,750 | test | INFO | pool-25-thread-19 | [table_view:display:72] Keys failed in bucket2:xV1RBonbPrQiAa-26-782000:1UlbM-26-889000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1373 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1275 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1365 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:14,609 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 54.69 2021-05-17 09:59:19,713 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 58.29 2021-05-17 09:59:24,815 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.67 2021-05-17 09:59:29,913 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 65.71 2021-05-17 09:59:35,023 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 09:59:40,104 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.84 2021-05-17 09:59:43,114 | test | INFO | pool-25-thread-4 | [table_view:display:72] Keys failed in default:lr-28-931000:xBtPzpimr0rlYknI-29-61000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1918 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1923 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1901 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1891 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:45,191 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 69.49 2021-05-17 09:59:45,278 | test | INFO | pool-25-thread-27 | [table_view:display:72] Keys failed in default:_default:mS8dht-8YLmX2Cil-35-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-615 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-677 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:47,253 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in default:KC7V3U-35-66000:HhuI-nEiIHHjk-JPb8g-35-66000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:48,150 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:lr-28-931000:Y-29-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1263 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:48,476 | test | INFO | pool-25-thread-15 | [table_view:display:72] Keys failed in default:KC7V3U-35-66000:T%ql-35-68000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-933 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-976 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-973 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:50,270 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 71.51 2021-05-17 09:59:50,464 | test | INFO | pool-25-thread-19 | [table_view:display:72] Keys failed in default:scope2:dFMhySeLe%upZJV-35-62000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-716 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-875 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:52,039 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:lr-28-931000:WQvSRX78QpiEJ4hpgrffe-29-60000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1137 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:52,624 | test | INFO | pool-25-thread-23 | [table_view:display:72] Keys failed in default:_default:4D2Cd4QI9Q8TM-35-57000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:55,375 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.45 2021-05-17 09:59:56,191 | test | INFO | pool-25-thread-30 | [table_view:display:72] Keys failed in default:_default:2NjAco5%-35-57000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-920 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 09:59:59,181 | test | INFO | pool-25-thread-25 | [table_view:display:72] Keys failed in default:_default:PSeQ-35-56000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:00,454 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.23 2021-05-17 10:00:00,999 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:lr-28-931000:Y-29-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1588 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1458 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1492 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:01,746 | test | INFO | pool-25-thread-27 | [table_view:display:72] Keys failed in default:_default:mS8dht-8YLmX2Cil-35-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:01,868 | test | INFO | pool-25-thread-19 | [table_view:display:72] Keys failed in default:scope2:dFMhySeLe%upZJV-35-62000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-920 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:04,214 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:lr-28-931000:WQvSRX78QpiEJ4hpgrffe-29-60000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:05,542 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 77.47 2021-05-17 10:00:05,632 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in default:_default:5v-35-56000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:09,910 | test | INFO | pool-25-thread-23 | [table_view:display:72] Keys failed in default:_default:4D2Cd4QI9Q8TM-35-57000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:10,424 | test | INFO | pool-25-thread-25 | [table_view:display:72] Keys failed in default:_default:PSeQ-35-56000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:10,630 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.76 2021-05-17 10:00:13,061 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:lr-28-931000:Y-29-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1605 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1750 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1795 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:13,181 | test | INFO | pool-25-thread-27 | [table_view:display:72] Keys failed in default:_default:mS8dht-8YLmX2Cil-35-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-911 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-942 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:14,056 | test | INFO | pool-25-thread-26 | [table_view:display:72] Keys failed in default:qg%qcTrNcT-29-549000:imh8O-pwXrqlE3XeXsFE-53-292000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1997 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:14,058 | test | INFO | pool-25-thread-26 | [table_view:display:72] Keys failed in default:qg%qcTrNcT-29-549000:imh8O-pwXrqlE3XeXsFE-53-292000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1997 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:14,884 | test | INFO | pool-25-thread-6 | [table_view:display:72] Keys failed in default:_default:Oqi3-z%UQTt0wDBbFnd-35-55000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-942 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:15,710 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:lr-28-931000:WQvSRX78QpiEJ4hpgrffe-29-60000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1430 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:15,711 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 81.54 2021-05-17 10:00:20,568 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in default:_default:5v-35-56000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:20,805 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 83.26 2021-05-17 10:00:22,391 | test | INFO | pool-25-thread-24 | [table_view:display:72] Keys failed in default:qg%qcTrNcT-29-549000:PmnwOxGP6C6ZhcN9FKVI-29-698000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:23,278 | test | INFO | pool-25-thread-7 | [table_view:display:72] Keys failed in default:scope1:G%Mlc_sAaEO-35-58000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:00:25,894 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 85.04 2021-05-17 10:00:30,967 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 86.76 2021-05-17 10:00:36,065 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 88.54 2021-05-17 10:00:41,213 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 90.89 2021-05-17 10:00:46,364 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 92.5 2021-05-17 10:00:51,447 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 94.27 2021-05-17 10:00:56,568 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 96.26 2021-05-17 10:01:01,680 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 98.03 2021-05-17 10:01:07,273 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 10:01:17,390 | test | INFO | pool-25-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 10:01:17,436 | test | INFO | pool-25-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 209.543000221 sec 2021-05-17 10:04:04,289 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 10:04:04,290 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 10:04:04,296 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 10:04:04,298 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 10:04:04,299 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 10:04:04,303 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 6 docs from 1000 to 1960 2021-05-17 10:04:04,305 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 9 docs from 1000 to 1960 2021-05-17 10:04:04,309 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 10:04:04,311 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 4 docs from 500 to 980 2021-05-17 10:05:33,221 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 3 docs from 500 to 980 2021-05-17 10:05:33,223 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 10:05:33,224 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 8 docs from 980 to 1440 2021-05-17 10:05:33,226 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 17 docs from 1000 to 1960 2021-05-17 10:05:33,226 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 7 docs from 980 to 1440 2021-05-17 10:05:33,226 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 10:05:33,229 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-17 10:05:33,229 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 12 docs from 1000 to 1960 2021-05-17 10:05:33,230 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 10:05:33,233 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 10:05:33,233 | test | ERROR | pool-25-thread-18 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 10:13:32,961 | test | INFO | pool-25-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******.****............*........................................***X | 96.353 | | 23.402 | ...........X | 23.402 | | 0.000 | ************.....................*................................***X | 97.450 | | 77.868 | ..........................................X | 77.868 | | 0.000 | *********.................................*****.***X | 62.233 | | 60.817 | ................................X | 60.817 | | 57.594 | ..............................****...............................................*...........***X | 161.600 | | 55.501 | .............................****..*...****.*.***X | 68.014 | | 63.810 | ..................................*.....*****.***X | 74.190 | | 68.203 | ....................................X | 68.203 | | 63.012 | .................................****.*....*****.................*.*****..............*****X | 129.582 | | 102.511 | .......................................................****X | 102.511 | | 99.079 | .....................................................*..*****.........****X | 117.923 | | 85.490 | ..............................................X | 85.490 | | 76.690 | .........................................****X | 76.690 | | 65.398 | ...................................X | 65.398 | | 62.995 | .................................***X | 62.995 | | 59.898 | ................................X | 59.898 | | 47.099 | .........................*****......****X | 58.896 | | 12.899 | ......X | 12.899 | | 0.800 | ****X | 0.800 | | 0.400 | ******......*....*****..........................................................................................****X | 180.596 | | 125.313 | ....................................................................X | 125.313 | | 113.307 | .............................................................****X | 113.307 | | 93.296 | ..................................................*****X | 93.296 | | 89.200 | ................................................****X | 89.200 | | 77.986 | ..........................................X | 77.986 | | 75.385 | ........................................*****.......*.*****.............................****X | 141.287 | | 91.610 | .................................................X | 91.610 | | 91.313 | .................................................***X | 91.313 | | 91.313 | .................................................X | 91.313 | | 65.196 | ...................................******......*..****.............*........****X | 117.602 | | 97.101 | ....................................................X | 97.101 | | 87.601 | ...............................................****X | 87.601 | | 73.293 | .......................................******.....X | 81.497 | | 79.500 | ...........................................*****....*****..................*....****X | 126.501 | | 84.402 | .............................................****X | 84.402 | | 83.103 | .............................................X | 83.103 | | 70.802 | ......................................*****....*.**X | 80.298 | | 80.298 | ...........................................**....X | 88.391 | | 88.290 | ...............................................****X | 88.290 | | 79.699 | ...........................................******......................*....***X | 128.118 | | 109.798 | ...........................................................X | 109.798 | | 86.595 | ..............................................****X | 86.595 | | 80.485 | ...........................................X | 80.485 | | 57.814 | ...............................****X | 57.814 | | 0.600 | ****X | 0.600 | | 0.600 | **********X | 0.800 | | 0.000 | **********X | 1.200 | | 0.200 | *****.....*......*****...........................*.......****X | 82.401 | | 18.498 | .........X | 18.498 | | 2.500 | ****X | 2.500 | | 2.000 | ****X | 2.000 | | 1.200 | X | 1.200 | | 1.200 | *****.****X | 2.600 | | 0.800 | *****X | 0.800 | | 0.000 | *****....................******......****X | 46.978 | | 39.278 | ....................X | 39.278 | | 1.799 | ****X | 1.799 | | 0.200 | ******.*****X | 2.399 | | 1.999 | ******.........................................*****X | 77.646 | | 4.497 | .*****.............*.......*****............*****X | 61.556 | | 26.455 | .............X | 26.455 | | 2.200 | ***X | 2.200 | | 1.900 | X | 1.900 | | 1.600 | ****X | 1.600 | | 0.600 | ************.****X | 2.100 | | 1.500 | X | 1.500 | | 1.400 | *****........*...................................****X | 78.959 | | 4.599 | .*****X | 4.599 | | 1.700 | **********X | 1.700 | | 1.399 | *****X | 1.399 | | 0.600 | ***********.***X | 2.000 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:13:34,167 | test | INFO | pool-25-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******.............*.........................................***X | 156.167 | | 113.907 | .......................................X | 113.907 | | 100.549 | ..................................***...........*............****..............****...........*.................***X | 284.026 | | 275.956 | ................................................................................................X | 275.956 | | 266.657 | ............................................................................................***X | 266.657 | | 85.534 | .............................*****.............****...............*......****...................*....***X | 248.636 | | 219.278 | ............................................................................****X | 219.278 | | 148.399 | ...................................................****.....*.***X | 166.103 | | 78.407 | ..........................X | 78.407 | | 52.391 | .................****.*.***X | 58.810 | | 52.412 | .................X | 52.412 | | 46.296 | ...............****...****X | 55.091 | | 47.003 | ...............X | 47.003 | | 45.710 | ...............****.******.....****X | 65.103 | | 58.492 | ...................X | 58.492 | | 58.093 | ...................****X | 58.093 | | 32.304 | ..........****X | 32.304 | | 19.099 | .....X | 19.099 | | 0.000 | **********................*....****X | 58.393 | | 47.899 | ...............X | 47.899 | | 44.598 | ..............****...*.....****X | 66.400 | | 64.299 | .....................X | 64.299 | | 62.196 | ....................****..*......*****..*****X | 89.307 | | 78.203 | ..........................****X | 78.203 | | 34.901 | ...........X | 34.901 | | 22.900 | .......****X | 22.900 | | 0.200 | ****X | 0.200 | | 0.200 | *****X | 0.200 | | 0.000 | ***********************............................*......***X | 99.114 | | 59.402 | ...................X | 59.402 | | 1.700 | ****X | 1.700 | | 1.500 | X | 1.500 | | 0.000 | *****.............*.....................***X | 97.602 | | 97.602 | .................................X | 97.602 | | 0.200 | *****X | 0.200 | | 0.000 | ****************X | 1.000 | | 0.200 | **********X | 2.200 | | 0.000 | ****************X | 0.400 | | 0.300 | X | 0.300 | | 0.100 | ***********........*......*****.......****X | 61.844 | | 35.777 | ...........X | 35.777 | | 0.000 | *****.................................****X | 95.296 | | 69.879 | .......................X | 69.879 | | 62.777 | .....................****X | 62.777 | | 37.011 | ............*****X | 37.011 | | 10.600 | ..X | 10.600 | | 3.500 | ****X | 3.500 | | 0.000 | **********************.******...........*...****..*.........***X | 75.099 | | 67.202 | ......................X | 67.202 | | 64.704 | .....................****X | 64.704 | | 0.000 | *****..............*....****X | 52.595 | | 51.164 | .................X | 51.164 | | 49.873 | ................*****....****X | 61.147 | | 52.764 | .................X | 52.764 | | 37.664 | ............****X | 37.664 | | 8.601 | ..X | 8.601 | | 1.200 | ****X | 1.200 | | 0.400 | *****X | 0.400 | | 0.000 | ******.*****X | 3.398 | | 1.200 | ****X | 1.200 | | 0.000 | ************.*****X | 3.799 | | 1.000 | ****X | 1.000 | | 0.700 | X | 0.700 | | 0.600 | *****.......................*.....****X | 81.946 | | 33.351 | ..........X | 33.351 | | 1.800 | ****X | 1.800 | | 0.000 | ***********........*...........................****X | 99.656 | | 97.156 | .................................X | 97.156 | | 95.450 | ................................****X | 95.450 | | 70.757 | .......................*****X | 70.757 | | 1.600 | **********X | 1.801 | | 1.800 | ***X | 1.800 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:13:35,391 | test | INFO | pool-25-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******........................*............................................................................***X | 1590.909 | | 819.167 | ..................................................X | 819.167 | | 535.514 | ................................**X | 535.514 | | 206.167 | ...........X | 206.167 | | 152.611 | ........****...*****...........***X | 368.867 | | 368.852 | ......................****X | 381.050 | | 145.662 | ........*****.........................****...........****X | 719.026 | | 693.248 | ..........................................X | 693.248 | | 684.024 | .........................................****X | 684.024 | | 304.964 | ..................***X | 304.964 | | 288.675 | .................X | 288.675 | | 278.905 | ................****..*****........**X | 433.692 | | 433.692 | ..........................**.......****X | 549.704 | | 528.862 | ................................****X | 528.862 | | 191.674 | ...........**********.*****...*........****X | 395.315 | | 351.948 | .....................X | 351.948 | | 339.155 | ....................*****..........*****X | 506.551 | | 400.721 | ........................**X | 400.721 | | 400.721 | ........................**......*****X | 503.695 | | 235.165 | .............*****..*....****X | 329.177 | | 201.970 | ...........X | 201.970 | | 169.301 | .........****..*....*.****...*.....*****....*X | 477.039 | | 477.039 | ............................**X | 477.039 | | 326.489 | ...................X | 326.489 | | 310.613 | ..................*****.****X | 324.891 | | 304.678 | ..................*****X | 315.621 | | 309.487 | ..................X | 309.487 | | 305.287 | ..................*****.....****X | 384.801 | | 337.437 | ....................X | 337.437 | | 328.831 | ...................*****......*****.*.*****.****X | 474.767 | | 470.438 | ............................******..*****X | 508.789 | | 224.503 | .............******..*****.....*.****X | 365.163 | | 296.098 | .................*****X | 296.098 | | 293.016 | .................*..***X | 321.705 | | 260.407 | ...............X | 260.407 | | 238.604 | .............*****...*......****X | 367.307 | | 284.606 | ................*****X | 284.606 | | 268.794 | ...............*****X | 268.794 | | 257.271 | ...............*****..*..X | 319.399 | | 319.399 | ...................********X | 319.542 | | 313.305 | ..................X | 313.305 | | 305.482 | ..................****X | 305.482 | | 290.351 | .................X | 290.351 | | 287.382 | .................*****.............****X | 499.826 | | 299.986 | .................****X | 299.986 | | 288.710 | .................X | 288.710 | | 260.803 | ...............*****.*****..*.........****X | 448.791 | | 405.128 | ........................****X | 405.128 | | 297.078 | .................X | 297.078 | | 262.079 | ...............****X | 262.079 | | 228.605 | .............*****.............*....****X | 501.361 | | 462.028 | ............................*.****X | 477.940 | | 393.383 | .......................X | 393.383 | | 348.668 | ....................****.X | 350.741 | | 313.596 | ..................*****.*****...............****X | 563.381 | | 511.070 | ...............................X | 511.070 | | 250.578 | ..............****X | 250.578 | | 246.491 | ..............*****...*.*****..******........****X | 470.710 | | 416.830 | .........................X | 416.830 | | 347.465 | ....................*****.X | 350.813 | | 343.441 | ....................*****...*....****X | 448.109 | | 409.853 | ........................X | 409.853 | | 393.897 | .......................****X | 393.897 | | 318.851 | ...................*****...*...........****X | 554.748 | | 509.765 | ...............................X | 509.765 | | 329.184 | ...................*****...........*****X | 495.507 | | 479.881 | .............................****X | 479.881 | | 455.255 | ...........................X | 455.255 | | 323.873 | ...................*****.......*..****X | 470.670 | | 354.613 | .....................******..........*...***X | 565.473 | | 389.710 | .......................X | 389.710 | | 342.508 | ....................*****..*****X | 375.197 | | 359.359 | .....................*****.....*...............****X | 677.942 | | 634.724 | ......................................X | 634.724 | | 628.484 | ......................................*****.......****X | 739.065 | +---------+-----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 10:13:35,500 | test | WARNING | Thread-81482 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:v-RTOyeJDWbUFs8Az-51-469000:QjsFGKwF-51-569000 create failed keys from task: [u'test_collections-2184', u'test_collections-2014'] 2021-05-17 10:13:35,500 | test | WARNING | Thread-81485 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:JJPZ-51-166000:nbTaH-MIdH-jMTd5TuM-51-317000 create failed keys from task: [u'test_collections-2184', u'test_collections-2014'] 2021-05-17 10:13:35,506 | test | WARNING | Thread-81549 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:xV1RBonbPrQiAa-26-782000:1UlbM-26-889000 update failed keys from task: [u'test_collections-1093', u'test_collections-1103', u'test_collections-1373', u'test_collections-1143', u'test_collections-1275', u'test_collections-1045', u'test_collections-1320', u'test_collections-1365', u'test_collections-1146'] 2021-05-17 10:13:35,509 | test | WARNING | Thread-81559 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:mS8dht-8YLmX2Cil-35-55000 delete failed keys from task: [u'test_collections-907', u'test_collections-823', u'test_collections-942', u'test_collections-866', u'test_collections-740', u'test_collections-719', u'test_collections-759', u'test_collections-911', u'test_collections-615', u'test_collections-954', u'test_collections-632', u'test_collections-677', u'test_collections-793'] 2021-05-17 10:13:35,513 | test | WARNING | Thread-81574 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Oqi3-z%UQTt0wDBbFnd-35-55000 delete failed keys from task: [u'test_collections-907', u'test_collections-942'] 2021-05-17 10:13:35,513 | test | WARNING | Thread-81575 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:PSeQ-35-56000 update failed keys from task: [u'test_collections-1190', u'test_collections-1235', u'test_collections-1269', u'test_collections-1401', u'test_collections-1423', u'test_collections-1336', u'test_collections-1382', u'test_collections-1394'] 2021-05-17 10:13:35,513 | test | WARNING | Thread-81579 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:4D2Cd4QI9Q8TM-35-57000 update failed keys from task: [u'test_collections-1190', u'test_collections-1401', u'test_collections-1423', u'test_collections-1325', u'test_collections-1360', u'test_collections-1394', u'test_collections-1266'] 2021-05-17 10:13:35,516 | test | WARNING | Thread-81591 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:5v-35-56000 update failed keys from task: [u'test_collections-988', u'test_collections-1342'] 2021-05-17 10:13:35,517 | test | WARNING | Thread-81593 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:qg%qcTrNcT-29-549000:PmnwOxGP6C6ZhcN9FKVI-29-698000 update failed keys from task: [u'test_collections-1031'] 2021-05-17 10:13:35,519 | test | WARNING | Thread-81594 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:qg%qcTrNcT-29-549000:imh8O-pwXrqlE3XeXsFE-53-292000 create failed keys from task: [u'test_collections-1997'] 2021-05-17 10:13:35,519 | test | WARNING | Thread-81562 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2NjAco5%-35-57000 delete failed keys from task: [u'test_collections-925', u'test_collections-947', u'test_collections-920', u'test_collections-951'] 2021-05-17 10:13:35,563 | test | WARNING | Thread-81640 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:G%Mlc_sAaEO-35-58000 update failed keys from task: [u'test_collections-982', u'test_collections-1031'] 2021-05-17 10:13:35,571 | test | WARNING | Thread-81484 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:JJPZ-51-166000:3_-51-316000 create failed keys from task: [u'test_collections-2184'] 2021-05-17 10:13:35,582 | test | WARNING | Thread-81551 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:tSRhw-wd2s-50-866000:yITBA-51-17000 create failed keys from task: [u'test_collections-2184', u'test_collections-2014'] 2021-05-17 10:13:35,604 | test | WARNING | Thread-81630 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:lr-28-931000:WQvSRX78QpiEJ4hpgrffe-29-60000 update failed keys from task: [u'test_collections-1190', u'test_collections-1109', u'test_collections-1000', u'test_collections-1387', u'test_collections-1137', u'test_collections-1336', u'test_collections-1217', u'test_collections-1230', u'test_collections-1252', u'test_collections-1099', u'test_collections-1155', u'test_collections-1430'] 2021-05-17 10:13:35,612 | test | WARNING | Thread-81631 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:lr-28-931000:Y-29-857000 update failed keys from task: [u'test_collections-1391', u'test_collections-1444', u'test_collections-1588', u'test_collections-1502', u'test_collections-1605', u'test_collections-1263', u'test_collections-1244', u'test_collections-1266', u'test_collections-1201', u'test_collections-1379', u'test_collections-1458', u'test_collections-1656', u'test_collections-1339', u'test_collections-1418', u'test_collections-1492', u'test_collections-1750', u'test_collections-1795'] 2021-05-17 10:13:35,619 | test | WARNING | Thread-81703 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KC7V3U-35-66000:T%ql-35-68000 delete failed keys from task: [u'test_collections-933', u'test_collections-976', u'test_collections-973'] 2021-05-17 10:13:35,627 | test | WARNING | Thread-81710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KC7V3U-35-66000:HhuI-nEiIHHjk-JPb8g-35-66000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 10:13:35,625 | test | WARNING | Thread-81700 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:dFMhySeLe%upZJV-35-62000 delete failed keys from task: [u'test_collections-716', u'test_collections-947', u'test_collections-920', u'test_collections-875', u'test_collections-780'] 2021-05-17 10:13:35,630 | test | WARNING | Thread-81633 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:lr-28-931000:xBtPzpimr0rlYknI-29-61000 update failed keys from task: [u'test_collections-1918', u'test_collections-1888', u'test_collections-1944', u'test_collections-1923', u'test_collections-1901', u'test_collections-1891'] 2021-05-17 10:13:49,549 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 10:15:07,124 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 10:15:07,125 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 10:15:25,161 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 10:15:25,161 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 10:15:43,934 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 10:15:43,936 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 10:16:47,848 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1289999485 seconds to complete 2021-05-17 10:16:47,864 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3394.868s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 23 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_23 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_24 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'recovery_type': 'delta', 'step_count': '1', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_24', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 24, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 10:16:57,259 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #24 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 10:16:57,940 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #24 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 10:16:58,358 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xdaf2b9 type=client open_count=1 channel=[id: 0xb568c030, 0.0.0.0/0.0.0.0:35958] timeout=300.0>) 2021-05-17 10:17:19,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xdaf2ba type=client open_count=1 channel=[id: 0x0762996e, 0.0.0.0/0.0.0.0:37143] timeout=300.0>) 2021-05-17 10:17:21,178 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xdaf2bb type=client open_count=1 channel=[id: 0x77d119c3, 0.0.0.0/0.0.0.0:48782] timeout=300.0>) 2021-05-17 10:17:23,197 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xdaf2bc type=client open_count=1 channel=[id: 0xa0bae3a4, 0.0.0.0/0.0.0.0:46233] timeout=300.0>) 2021-05-17 10:17:47,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xdaf2bd type=client open_count=1 channel=[id: 0x5268ae83, 0.0.0.0/0.0.0.0:45807] timeout=300.0>) 2021-05-17 10:17:49,505 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xdaf2be type=client open_count=1 channel=[id: 0xac0c0b4d, 0.0.0.0/0.0.0.0:33712] timeout=300.0>) 2021-05-17 10:17:51,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xdaf2bf type=client open_count=1 channel=[id: 0x252bda96, 0.0.0.0/0.0.0.0:43558] timeout=300.0>) 2021-05-17 10:18:15,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xdaf2c0 type=client open_count=1 channel=[id: 0x7a23f4f2, 0.0.0.0/0.0.0.0:56000] timeout=300.0>) 2021-05-17 10:18:17,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xdaf2c1 type=client open_count=1 channel=[id: 0x3fcf371c, 0.0.0.0/0.0.0.0:50164] timeout=300.0>) 2021-05-17 10:18:19,471 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xdaf2c2 type=client open_count=1 channel=[id: 0x62f9d1f8, 0.0.0.0/0.0.0.0:54035] timeout=300.0>) 2021-05-17 10:18:43,694 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xdaf2c3 type=client open_count=1 channel=[id: 0x74364593, 0.0.0.0/0.0.0.0:58905] timeout=300.0>) 2021-05-17 10:18:45,713 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xdaf2c4 type=client open_count=1 channel=[id: 0x2a550831, 0.0.0.0/0.0.0.0:59051] timeout=300.0>) 2021-05-17 10:18:47,730 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xdaf2c5 type=client open_count=1 channel=[id: 0xc37c14fa, 0.0.0.0/0.0.0.0:56510] timeout=300.0>) 2021-05-17 10:19:12,160 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xdaf2c6 type=client open_count=1 channel=[id: 0x84bd617f, 0.0.0.0/0.0.0.0:48467] timeout=300.0>) 2021-05-17 10:19:14,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xdaf2c7 type=client open_count=1 channel=[id: 0x82bcabcd, 0.0.0.0/0.0.0.0:44821] timeout=300.0>) 2021-05-17 10:19:16,197 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 10:19:33,246 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 10:19:47,019 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 10:19:47,042 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #24 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 10:19:47,079 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #24 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 10:20:00,369 | test | INFO | pool-26-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 9.5315078082 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 10:20:00,394 | test | INFO | pool-26-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:20:10,471 | test | INFO | pool-26-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 10:20:10,497 | test | INFO | pool-26-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.128000021 sec 2021-05-17 10:20:10,773 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 10:21:01,716 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 10:21:02,068 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 579391000 | 97638991 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 613541808 | 115627076 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 768450680 | 59566866 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 10:21:02,068 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 10:21:46,280 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 10:21:46,298 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 10:37:43,878 | test | INFO | pool-26-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................................................................................................****X | 123.044 | | 8.185 | .....X | 8.185 | | 0.000 | **********************************......*****....*****X | 13.507 | | 6.403 | ....******.......****X | 15.005 | | 10.002 | .......X | 10.002 | | 7.502 | .....****X | 7.502 | | 0.499 | X | 0.499 | | 0.000 | **********************************.........*****X | 12.003 | | 6.001 | ...****X | 6.001 | | 3.000 | .X | 3.000 | | 1.500 | *****..*****X | 4.501 | | 1.500 | ****X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | ****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | **********.*.****X | 3.001 | | 0.000 | ***********..****X | 3.001 | | 1.001 | X | 1.001 | | 0.000 | *****...*****X | 4.501 | | 1.500 | ******..*****X | 4.501 | | 0.000 | *****..*.****X | 4.501 | | 1.500 | X | 1.500 | | 0.000 | ***********.****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ***********..******..*****X | 6.101 | | 5.901 | ...****X | 5.901 | | 5.001 | ...X | 5.001 | | 1.500 | ****X | 1.500 | | 0.000 | ******...****X | 4.500 | | 1.500 | X | 1.500 | | 0.000 | *********************************************************.........*****X | 11.903 | | 0.000 | *****.*.....*...***X | 11.200 | | 4.000 | ..X | 4.000 | | 0.800 | ****X | 0.800 | | 0.000 | ******...**********X | 4.500 | | 1.500 | X | 1.500 | | 0.000 | *****************.****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *****.*.***X | 3.000 | | 3.000 | .****X | 3.000 | | 2.200 | X | 2.200 | | 0.600 | X | 0.600 | | 0.000 | *****..*.***X | 4.503 | | 2.400 | X | 2.400 | | 0.000 | **********************.****X | 1.500 | | 0.000 | ******************.****X | 1.500 | | 0.000 | ***********.*..****X | 4.500 | | 2.400 | X | 2.400 | | 0.000 | ***********.****X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | ****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *******************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:37:44,499 | test | INFO | pool-26-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******...................................................................*****...........................*......***X | 962.188 | | 915.576 | ..............................................................................................X | 915.576 | | 679.272 | .....................................................................*****.........*.****X | 778.518 | | 745.014 | ............................................................................X | 745.014 | | 724.662 | ..........................................................................****.**...***X | 766.977 | | 741.946 | ............................................................................X | 741.946 | | 709.712 | ........................................................................****X | 709.712 | | 593.945 | ............................................................X | 593.945 | | 533.448 | ......................................................***X | 533.448 | | 526.675 | .....................................................X | 526.675 | | 472.247 | ................................................****X | 472.247 | | 431.591 | ...........................................*.*****...........*..****X | 560.018 | | 534.351 | ......................................................******......*****X | 586.972 | | 511.425 | ....................................................*****..*.****X | 545.787 | | 534.857 | ......................................................*.****X | 541.748 | | 539.378 | .......................................................X | 539.378 | | 515.628 | ....................................................****X | 515.628 | | 499.053 | ..................................................******.....X | 543.936 | | 542.625 | .......................................................****...*.....***X | 616.168 | | 595.926 | ............................................................X | 595.926 | | 490.166 | .................................................*****.X | 491.989 | | 468.808 | ...............................................****X | 468.808 | | 391.726 | .......................................*****.....*........*****.......*.****X | 590.388 | | 494.748 | ..................................................******....****X | 536.814 | | 529.016 | .....................................................X | 529.016 | | 505.848 | ...................................................*****...*..****X | 550.328 | | 538.041 | ......................................................****X | 538.041 | | 523.090 | .....................................................*..****X | 547.647 | | 504.342 | ...................................................X | 504.342 | | 453.398 | ..............................................*****....****X | 496.733 | | 487.111 | .................................................*....****X | 521.080 | | 509.209 | ...................................................X | 509.209 | | 461.773 | ..............................................****X | 461.773 | | 460.057 | ..............................................*..****X | 477.408 | | 450.324 | .............................................*****X | 450.324 | | 420.487 | ..........................................*****........*....****X | 531.290 | | 512.394 | ....................................................X | 512.394 | | 472.857 | ................................................****X | 472.857 | | 455.098 | ..............................................******..******.*****..*.***X | 514.356 | | 502.529 | ...................................................******....*****X | 543.292 | | 525.191 | .....................................................*****X | 525.191 | | 375.730 | ......................................*****.......*....****X | 486.594 | | 466.098 | ...............................................*****X | 466.098 | | 451.325 | .............................................*.****..X | 477.052 | | 472.634 | ................................................*****.....*....****X | 564.344 | | 562.180 | .........................................................X | 562.180 | | 560.075 | .........................................................****X | 560.075 | | 532.433 | ......................................................X | 532.433 | | 504.514 | ...................................................*****......*..****X | 586.088 | | 530.277 | ......................................................****X | 530.277 | | 525.177 | .....................................................X | 525.177 | | 482.873 | .................................................*****.....*****X | 538.332 | | 448.898 | .............................................******.......*......***X | 573.077 | | 518.912 | ....................................................X | 518.912 | | 475.412 | ................................................*****..*...****X | 524.299 | | 449.814 | .............................................X | 449.814 | | 393.708 | .......................................****..*....*****.......X | 514.531 | | 501.828 | ...................................................****X | 501.828 | | 424.587 | ...........................................X | 424.587 | | 382.148 | ......................................****...*............****X | 524.735 | | 459.762 | ..............................................X | 459.762 | | 452.490 | ..............................................****..*..............*****.X | 617.047 | | 559.948 | .........................................................****X | 559.948 | | 557.047 | ........................................................X | 557.047 | | 515.082 | ....................................................****X | 515.082 | | 501.917 | ...................................................****X | 509.350 | | 488.893 | .................................................X | 488.893 | | 460.891 | ..............................................******.....****X | 502.323 | | 416.416 | ..........................................X | 416.416 | | 384.308 | ......................................*****.....................*****..*****X | 602.708 | | 559.422 | .........................................................****X | 559.422 | | 551.359 | ........................................................X | 551.359 | | 498.820 | ..................................................*****....*.***X | 545.709 | | 523.485 | .....................................................X | 523.485 | | 413.049 | .........................................*****....*.......****X | 511.499 | | 507.339 | ...................................................*.****...X | 543.642 | | 504.690 | ...................................................****X | 504.690 | | 442.600 | ............................................*.****..*...*****X | 497.202 | | 430.695 | ...........................................*****...X | 457.071 | | 447.570 | .............................................***X | 447.570 | | 443.065 | .............................................*...*.....****..X | 547.602 | | 546.978 | .......................................................*****.*..*****...****X | 601.135 | | 592.760 | ............................................................X | 592.760 | | 561.878 | .........................................................****X | 561.878 | | 529.492 | ......................................................*..***X | 554.953 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:37:45,091 | test | INFO | pool-26-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******.............................................................****X | 12.004 | | 0.799 | ...X | 0.799 | | 0.000 | ***************************..*.*****.*****..*.****X | 1.501 | | 0.000 | ***********..........*....****X | 2.901 | | 0.100 | X | 0.100 | | 0.000 | *****.......****X | 1.500 | | 1.100 | ....X | 1.100 | | 0.000 | ***********************........................*****...............................*.............*****.****X | 13.504 | | 13.503 | ....................................................................X | 13.503 | | 13.503 | ....................................................................****X | 13.503 | | 12.502 | ...............................................................X | 12.502 | | 12.002 | ............................................................****X | 12.002 | | 0.000 | ******.......****X | 1.501 | | 1.500 | ......X | 1.500 | | 1.500 | ......*****...............*.****X | 4.502 | | 0.000 | ***********..............................******..............*****X | 8.600 | | 4.901 | ........................****X | 4.901 | | 4.302 | .....................*.*****.........................................*.....................****X | 16.505 | | 6.501 | ................................X | 6.501 | | 1.600 | .......*****............................................................................................*****X | 19.405 | | 0.000 | ***********.......................*****X | 4.501 | | 0.000 | ***********....................*..........****X | 6.001 | | 5.701 | ............................X | 5.701 | | 4.100 | ....................****X | 4.100 | | 0.400 | .****X | 0.400 | | 0.000 | ************.............................................................*...............................****X | 18.001 | | 13.002 | ..................................................................X | 13.002 | | 10.501 | .....................................................****X | 10.501 | | 1.500 | ......*****X | 1.500 | | 0.000 | ***********.....*..**********X | 1.500 | | 0.000 | *****************........*.....****..............*...........................................................****X | 16.804 | | 14.404 | .........................................................................X | 14.404 | | 4.501 | ......................*****.....*****X | 5.600 | | 1.900 | ........******......*****...............*........*****.....................*..........****X | 13.501 | | 6.000 | .............................****X | 6.000 | | 4.400 | .....................X | 4.400 | | 2.000 | .........X | 2.000 | | 0.000 | ****.......*........****X | 3.000 | | 2.200 | ..........*****......X | 3.400 | | 2.800 | .............X | 2.800 | | 2.300 | ..........***X | 2.300 | | 1.100 | ....X | 1.100 | | 0.000 | *****.*....****..........*.............*.....***X | 6.496 | | 5.396 | ..........................X | 5.396 | | 5.102 | .........................X | 5.102 | | 4.503 | ......................***X | 4.503 | | 1.500 | ......X | 1.500 | | 0.000 | ****....*...........****X | 3.001 | | 2.201 | ..........X | 2.201 | | 1.100 | ....*****.X | 1.200 | | 0.400 | .*****.....*........***X | 3.000 | | 2.200 | ..........X | 2.200 | | 0.000 | **********....*...*........***X | 3.000 | | 2.600 | ............X | 2.600 | | 1.500 | ......****X | 1.500 | | 0.500 | .X | 0.500 | | 0.000 | ****************...*..*..****...........*.....*****.....*..****X | 6.000 | | 3.000 | ..............X | 3.000 | | 1.500 | ......***X | 1.500 | | 1.100 | ....X | 1.100 | | 0.500 | .X | 0.500 | | 0.000 | *******************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 10:37:45,355 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 74.8687434372 | 6437847040 | 4349124608 | 2405462016 / 7511588864 | 101595 / 206447 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 69.1393620213 | 6437847040 | 4276805632 | 2421952512 / 7511588864 | 99812 / 200697 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 77.0197636376 | 6437847040 | 4268363776 | 2435010560 / 7511588864 | 100087 / 199943 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 63.5156080732 | 6437847040 | 3832893440 | 3066515456 / 9530753024 | 101454 / 198692 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 53.6161219463 | 6437847040 | 4272349184 | 2443415552 / 7511588864 | 101230 / 202418 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 10:37:58,737 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 10:39:26,111 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164590152 | 269344255 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162400656 | 271198229 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 443014760 | 452282189 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 10:39:26,114 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #24 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 10:39:36,256 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 10:39:36,257 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 10:39:36,259 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:765] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 10:39:47,283 | test | INFO | pool-26-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.62599992752 seconds to complete 2021-05-17 10:39:47,289 | test | INFO | pool-26-thread-23 | [common_lib:sleep:22] Sleep 8.62599992752 seconds. Reason: Wait after rebalance complete 2021-05-17 10:41:55,973 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-05-17 10:41:55,973 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 10:41:55,973 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 10:42:03,499 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 10:53:49,390 | test | INFO | pool-26-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..*..........*****......................*......................*****...*....****X | 163.133 | | 162.621 | ..............................................................X | 162.621 | | 151.360 | ..........................................................****X | 151.360 | | 140.572 | ......................................................*...*****...................*.****X | 200.024 | | 168.376 | ................................................................X | 168.376 | | 165.376 | ...............................................................*****...........X | 192.038 | | 190.838 | .........................................................................***X | 190.838 | | 174.848 | ...................................................................X | 174.848 | | 170.584 | .................................................................*****..........*........................****X | 255.191 | | 249.668 | ................................................................................................X | 249.668 | | 209.536 | .................................................................................****X | 209.536 | | 201.174 | .............................................................................X | 201.174 | | 177.789 | ....................................................................****X | 177.789 | | 163.074 | ..............................................................*......*****......X | 193.082 | | 175.429 | ...................................................................*****.*.....*.***X | 192.328 | | 192.107 | ..........................................................................*....*..***X | 209.246 | | 179.480 | .....................................................................X | 179.480 | | 112.340 | ...........................................*****..*....****X | 127.917 | | 84.763 | ................................X | 84.763 | | 68.564 | .........................****X | 68.564 | | 67.866 | .........................*.****X | 70.476 | | 59.677 | ......................X | 59.677 | | 43.878 | ................*****...*..****X | 58.480 | | 55.275 | ....................******...******...****X | 68.958 | | 60.597 | ......................X | 60.597 | | 52.999 | ...................*****.....*****X | 63.868 | | 16.695 | .....X | 16.695 | | 16.695 | .....*X | 16.695 | | 16.695 | .....*X | 16.695 | | 10.193 | ..X | 10.193 | | 1.399 | ****X | 1.399 | | 0.999 | X | 0.999 | | 0.800 | *********X | 2.498 | | 2.098 | X | 2.098 | | 1.799 | *****X | 1.799 | | 1.799 | ****X | 1.799 | | 1.798 | X | 1.798 | | 1.099 | *****X | 1.099 | | 0.800 | *******.***X | 3.399 | | 2.399 | *X | 2.399 | | 0.300 | **********.*X | 3.798 | | 3.798 | *X | 3.798 | | 3.798 | *X | 3.798 | | 1.500 | X | 1.500 | | 0.800 | *****.****X | 2.599 | | 2.299 | X | 2.299 | | 1.700 | X | 1.700 | | 1.300 | ***X | 1.300 | | 0.800 | X | 0.800 | | 0.300 | ******.*****X | 3.000 | | 2.999 | *****X | 2.999 | | 2.517 | ****X | 2.517 | | 1.500 | X | 1.500 | | 1.000 | ****X | 1.000 | | 0.600 | X | 0.600 | | 0.500 | *****************X | 2.499 | | 2.300 | ****X | 2.300 | | 0.599 | X | 0.599 | | 0.399 | ******.****X | 3.196 | | 1.899 | ****X | 1.899 | | 1.898 | *****X | 2.098 | | 1.897 | X | 1.897 | | 0.401 | ****X | 0.401 | | 0.200 | ******X | 1.597 | | 1.299 | ****X | 1.299 | | 1.099 | *****X | 1.099 | | 1.099 | X | 1.099 | | 0.300 | *******.****X | 2.597 | | 2.297 | X | 2.297 | | 2.197 | ****X | 2.197 | | 1.698 | X | 1.698 | | 1.199 | **********X | 1.300 | | 0.599 | X | 0.599 | | 0.200 | ************.****X | 2.896 | | 1.796 | X | 1.796 | | 1.099 | **********X | 1.598 | | 0.698 | X | 0.698 | | 0.000 | X | 0.000 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:53:50,299 | test | INFO | pool-26-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.*****....*....................*****..................*...............................................****X | 728.672 | | 713.255 | .......................................................................................X | 713.255 | | 699.566 | .....................................................................................****X | 700.172 | | 637.957 | .............................................................................X | 637.957 | | 506.005 | .............................................................X | 506.005 | | 437.980 | .....................................................***********.....*****.....*...................................****X | 807.931 | | 806.197 | ..................................................................................................X | 806.197 | | 579.589 | ......................................................................****X | 579.589 | | 536.523 | .................................................................*..*..*X | 572.718 | | 572.718 | .....................................................................*X | 572.718 | | 572.718 | .....................................................................X | 572.718 | | 311.007 | .....................................******...*****X | 332.926 | | 281.312 | .................................*****..X | 292.686 | | 282.600 | .................................*****...................*.......*X | 484.982 | | 484.982 | ...........................................................**X | 484.982 | | 399.418 | ................................................X | 399.418 | | 373.930 | .............................................***X | 373.930 | | 358.599 | ...........................................X | 358.599 | | 321.577 | ......................................*****....*...........******.****X | 444.677 | | 409.181 | .................................................X | 409.181 | | 325.848 | .......................................*****...X | 353.047 | | 330.063 | .......................................*****.......*........****X | 450.074 | | 431.722 | ....................................................X | 431.722 | | 429.810 | ....................................................*****..................*....****X | 613.990 | | 523.090 | ...............................................................X | 523.090 | | 294.963 | ...................................*****..........******....................*........**X | 598.741 | | 598.741 | .........................................................................*X | 598.741 | | 346.517 | .........................................******.................****X | 483.646 | | 458.708 | .......................................................X | 458.708 | | 447.605 | ......................................................X | 447.605 | | 433.855 | ....................................................****.X | 436.655 | | 425.058 | ...................................................*.****.............*...****......*.......*.....****..*.......*..**X | 799.550 | | 799.550 | .................................................................................................X | 799.550 | | 777.631 | ...............................................................................................X | 777.631 | | 702.363 | .....................................................................................X | 702.363 | | 670.122 | .................................................................................X | 670.122 | | 670.122 | .................................................................................**X | 670.122 | | 624.686 | ............................................................................X | 624.686 | | 600.586 | .........................................................................X | 600.586 | | 600.586 | .........................................................................****....X | 635.212 | | 625.740 | ............................................................................***X | 625.740 | | 625.740 | ............................................................................*......*.........***X | 747.563 | | 733.480 | .........................................................................................X | 733.480 | | 613.794 | ..........................................................................*****.......*...****X | 694.611 | | 632.058 | .............................................................................X | 632.058 | | 585.622 | .......................................................................****X | 585.622 | | 500.993 | .............................................................*****..*..X | 538.606 | | 532.132 | ................................................................****...*..*.......***X | 626.423 | | 573.275 | .....................................................................X | 573.275 | | 526.162 | ................................................................*****.......*****X | 589.527 | | 488.518 | ...........................................................X | 488.518 | | 471.704 | .........................................................*****.....******................X | 646.034 | | 646.034 | ..............................................................................****X | 646.034 | | 536.062 | .................................................................*****..*.*...****X | 583.261 | | 525.569 | ................................................................******..****X | 542.911 | | 524.308 | ...............................................................X | 524.308 | | 479.541 | ..........................................................*****.......*..****X | 550.855 | | 543.063 | ..................................................................*..*****...X | 582.037 | | 567.139 | .....................................................................******....****X | 602.200 | | 579.920 | ......................................................................X | 579.920 | | 557.977 | ....................................................................****X | 557.977 | | 506.383 | .............................................................******........*****X | 569.399 | | 550.328 | ...................................................................******.......*....*****....*....****X | 707.545 | | 670.937 | ..................................................................................X | 670.937 | | 638.950 | ..............................................................................****X | 638.950 | | 599.608 | .........................................................................**.***X | 607.489 | | 600.642 | .........................................................................******.*....****X | 645.971 | | 623.486 | ............................................................................*......****X | 676.122 | | 672.002 | ..................................................................................X | 672.002 | | 656.117 | ................................................................................******..............*X | 768.768 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:53:51,190 | test | INFO | pool-26-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ************....*....*****..............*................*****......*****X | 76.231 | | 69.637 | .......................................*****...*......******.*****.*...******.....****X | 100.631 | | 96.931 | .......................................................X | 96.931 | | 89.926 | ...................................................******.........*****X | 104.620 | | 89.635 | ...................................................*****.........*......*****....X | 121.890 | | 117.584 | ...................................................................****X | 117.584 | | 114.356 | ..................................................................X | 114.356 | | 111.859 | ................................................................***X | 111.859 | | 107.149 | .............................................................X | 107.149 | | 98.629 | ........................................................****X | 98.629 | | 93.450 | .....................................................X | 93.450 | | 88.149 | ..................................................*****...X | 92.336 | | 90.941 | ....................................................*...***X | 97.129 | | 94.429 | ......................................................*.******.........X | 111.945 | | 111.159 | ................................................................****X | 111.159 | | 110.359 | ...............................................................X | 110.359 | | 106.750 | .............................................................***X | 106.750 | | 106.045 | .............................................................*..*****...X | 115.262 | | 113.043 | .................................................................*****...*...............****X | 144.636 | | 140.312 | .................................................................................*......****X | 151.306 | | 135.597 | ..............................................................................X | 135.597 | | 120.999 | .....................................................................*****....*.....*****...........*..*****........X | 170.464 | | 162.955 | ..............................................................................................****X | 162.955 | | 101.973 | ..........................................................X | 101.973 | | 84.769 | ................................................***X | 84.769 | | 74.169 | ..........................................X | 74.169 | | 41.884 | .......................****X | 41.884 | | 31.384 | .................X | 31.384 | | 12.001 | ......X | 12.001 | | 3.099 | ***X | 3.099 | | 2.699 | X | 2.699 | | 1.899 | ****X | 1.899 | | 1.798 | X | 1.798 | | 1.798 | X | 1.798 | | 1.698 | ****.***********X | 3.399 | | 0.900 | X | 0.900 | | 0.500 | ****X | 0.500 | | 0.300 | X | 0.300 | | 0.100 | *****..*****X | 4.297 | | 3.099 | *****X | 3.099 | | 2.999 | ****X | 2.999 | | 1.899 | X | 1.899 | | 0.500 | ****X | 0.500 | | 0.400 | ******.*******X | 3.399 | | 3.199 | ****X | 3.199 | | 1.269 | X | 1.269 | | 0.669 | *****.*****X | 2.400 | | 1.601 | X | 1.601 | | 1.500 | ****.*.****X | 4.599 | | 2.597 | X | 2.597 | | 1.701 | *****.*****X | 2.498 | | 0.798 | X | 0.798 | | 0.300 | *****..****X | 4.098 | | 3.495 | .X | 3.495 | | 2.100 | *****X | 2.100 | | 1.499 | *****X | 1.499 | | 1.099 | ******.****X | 2.999 | | 1.797 | X | 1.797 | | 1.199 | *****.*****X | 2.898 | | 2.697 | X | 2.697 | | 1.200 | ****X | 1.200 | | 1.200 | ******.*.****X | 3.596 | | 2.196 | *X | 2.196 | | 1.199 | **********.*************.***********X | 4.497 | | 3.997 | .X | 3.997 | | 2.998 | ****X | 2.998 | | 1.697 | X | 1.697 | | 0.999 | *X | 0.999 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 10:53:51,444 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 10:53:51,500 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 10:54:01,957 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 10:54:02,286 | test | INFO | pool-26-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.20666322228 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 45.0519965336 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 55.8324027933 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 61.8989683505 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 51.0674466277 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 10:54:02,335 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:54:07,444 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:54:08,729 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 10:54:13,596 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:54:18,690 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:54:24,075 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 10:54:29,438 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.42 2021-05-17 10:54:34,549 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 6.79 2021-05-17 10:54:38,755 | test | INFO | pool-26-thread-16 | [table_view:display:72] Keys failed in bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1115 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1085 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:54:39,711 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 14.52 2021-05-17 10:54:44,877 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 19.85 2021-05-17 10:54:47,240 | test | INFO | pool-26-thread-27 | [table_view:display:72] Keys failed in bucket1:X_D_RYQRe7bk-7-1000:b-iF5QGritfbmL2Vs-7-136000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:54:47,242 | test | INFO | pool-26-thread-27 | [table_view:display:72] Keys failed in bucket1:X_D_RYQRe7bk-7-1000:b-iF5QGritfbmL2Vs-7-136000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-526 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:54:50,019 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 23.0 2021-05-17 10:54:53,502 | test | INFO | pool-26-thread-16 | [table_view:display:72] Keys failed in bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:54:55,137 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 25.57 2021-05-17 10:55:00,207 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 27.62 2021-05-17 10:55:00,607 | test | INFO | pool-26-thread-2 | [table_view:display:72] Keys failed in bucket1:HuCELxXpvN4amFSUWI-6-316000:S-VkqImDKX-6-474000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:00,608 | test | INFO | pool-26-thread-2 | [table_view:display:72] Keys failed in bucket1:HuCELxXpvN4amFSUWI-6-316000:S-VkqImDKX-6-474000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-570 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:02,588 | test | INFO | pool-26-thread-7 | [table_view:display:72] Keys failed in bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:05,298 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 29.16 2021-05-17 10:55:07,618 | test | INFO | pool-26-thread-16 | [table_view:display:72] Keys failed in bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1507 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1452 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1497 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:10,382 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 36.9 2021-05-17 10:55:15,516 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 46.55 2021-05-17 10:55:20,588 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 49.8 2021-05-17 10:55:23,933 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket2:_default:Gwg3Y4iQ-10-818000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:25,325 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-756 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:25,683 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 54.6 2021-05-17 10:55:26,647 | test | INFO | pool-26-thread-7 | [table_view:display:72] Keys failed in bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:28,898 | test | INFO | pool-26-thread-15 | [table_view:display:72] Keys failed in bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1738 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1653 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1622 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1600 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1667 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1645 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1668 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1724 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1790 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1750 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1662 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1761 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1696 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:30,760 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 56.17 2021-05-17 10:55:35,835 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 58.79 2021-05-17 10:55:36,969 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:40,921 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 63.15 2021-05-17 10:55:43,401 | test | INFO | pool-26-thread-15 | [table_view:display:72] Keys failed in bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1848 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1929 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1930 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1833 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:46,028 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 74.78 2021-05-17 10:55:47,411 | test | INFO | pool-26-thread-22 | [table_view:display:72] Keys failed in default:Sh-10-856000:voncfT9jCflszxkM4FS-10-857000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:50,453 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in default:scope2:-j5VS-10-853000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:51,187 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 80.7 2021-05-17 10:55:56,286 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 82.58 2021-05-17 10:55:56,753 | test | INFO | pool-26-thread-25 | [table_view:display:72] Keys failed in default:Sh-10-856000:Clg--10-856000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1137 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:57,276 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in default:scope2:te2XeiST-10-852000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-933 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-973 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:58,466 | test | INFO | pool-26-thread-26 | [table_view:display:72] Keys failed in default:af1F9fO-Yko1IiTk-2-980000:E-3-114000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1901 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1891 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:55:59,395 | test | INFO | pool-26-thread-22 | [table_view:display:72] Keys failed in default:Sh-10-856000:voncfT9jCflszxkM4FS-10-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:01,365 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 84.36 2021-05-17 10:56:02,229 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in default:scope2:-j5VS-10-853000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:06,457 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 86.86 2021-05-17 10:56:07,398 | test | INFO | pool-26-thread-20 | [table_view:display:72] Keys failed in default:af1F9fO-Yko1IiTk-2-980000:E-3-114000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-465 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-563 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:11,562 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: running, progress: 88.81 2021-05-17 10:56:13,160 | test | INFO | pool-26-thread-25 | [table_view:display:72] Keys failed in default:Sh-10-856000:Clg--10-856000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:13,339 | test | INFO | pool-26-thread-21 | [table_view:display:72] Keys failed in default:_default:5TA95GMJEJfGmk8N_-10-846000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:13,342 | test | INFO | pool-26-thread-7 | [table_view:display:72] Keys failed in default:Sh-10-856000:vMDyuWfbAKeqYydl-10-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:16,216 | test | INFO | pool-26-thread-22 | [table_view:display:72] Keys failed in default:Sh-10-856000:voncfT9jCflszxkM4FS-10-857000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:16,223 | test | INFO | pool-26-thread-26 | [table_view:display:72] Keys failed in default:EkQo-10-853000:il9spZ--10-855000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:16,223 | test | INFO | pool-26-thread-30 | [table_view:display:72] Keys failed in default:_default:cq98M-nJI6Cf8nV4-10-846000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:16,609 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in default:scope2:-j5VS-10-853000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:20,368 | test | INFO | pool-26-thread-20 | [table_view:display:72] Keys failed in default:af1F9fO-Yko1IiTk-2-980000:E-3-114000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-713 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-734 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-672 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-683 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:21,707 | test | INFO | pool-26-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 10:56:21,760 | test | INFO | pool-26-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 139.474999905 sec 2021-05-17 10:56:23,035 | test | INFO | pool-26-thread-21 | [table_view:display:72] Keys failed in default:_default:5TA95GMJEJfGmk8N_-10-846000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:56:23,581 | test | INFO | pool-26-thread-8 | [table_view:display:72] Keys failed in default:_default:kCQV-v9mgbm-10-847000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 10:58:11,391 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 10:58:11,394 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 10:58:11,394 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 10:58:11,395 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 27 docs from 1000 to 1960 2021-05-17 10:58:11,395 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 4 docs from 0 to 1000 2021-05-17 10:58:11,397 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 15 docs from 1000 to 1960 2021-05-17 10:58:11,398 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 4 docs from 0 to 1000 2021-05-17 10:59:36,650 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 10 docs from 0 to 1000 2021-05-17 10:59:36,653 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 10:59:36,653 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 11:00:02,483 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 9 docs from 980 to 1440 2021-05-17 11:00:02,483 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 13 docs from 980 to 1440 2021-05-17 11:00:02,486 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 7 docs from 980 to 1440 2021-05-17 11:00:02,489 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 11:00:02,490 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 8 docs from 980 to 1440 2021-05-17 11:00:02,490 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 11:00:02,490 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 11:00:02,492 | test | ERROR | pool-26-thread-4 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 11:08:26,940 | test | INFO | pool-26-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..*.****.......*....***................................*..................................**X | 189.228 | | 76.259 | ...............................*****.....***X | 88.051 | | 80.607 | .................................X | 80.607 | | 76.594 | ...............................**X | 76.594 | | 68.555 | ............................X | 68.555 | | 36.380 | ..............***X | 36.380 | | 29.675 | ...........X | 29.675 | | 14.589 | .....****.................*.............................................................**X | 199.118 | | 153.695 | ................................................................X | 153.695 | | 50.678 | ....................****....................*****.......*****......*****......*...****....................*.................****X | 235.093 | | 159.129 | ..................................................................****X | 159.129 | | 140.792 | ..........................................................*.****X | 141.998 | | 137.495 | .........................................................X | 137.495 | | 130.498 | ......................................................*****...........*...****X | 164.007 | | 138.606 | .........................................................*****....*..*****..........................*****X | 211.590 | | 190.670 | ................................................................................X | 190.670 | | 172.420 | ........................................................................****.....X | 185.420 | | 183.515 | .............................................................................*****............*****.......X | 229.603 | | 220.412 | ............................................................................................*****...****X | 226.829 | | 177.096 | ..........................................................................X | 177.096 | | 101.997 | ..........................................*****...*****...............................*......................****X | 232.916 | | 113.205 | ...............................................*****X | 113.205 | | 86.099 | ...................................****X | 86.099 | | 60.090 | ........................X | 60.090 | | 42.898 | .................****X | 42.898 | | 2.900 | ****X | 2.900 | | 1.800 | X | 1.800 | | 1.700 | X | 1.700 | | 0.700 | ***************X | 1.999 | | 1.801 | *****X | 1.801 | | 0.800 | *****.........................****X | 59.404 | | 50.504 | ....................X | 50.504 | | 42.000 | ................****X | 42.000 | | 1.000 | ****X | 1.000 | | 0.500 | X | 0.500 | | 0.200 | *********X | 1.600 | | 0.900 | X | 0.900 | | 0.800 | **********X | 0.800 | | 0.800 | *****............*****....*.............*****X | 70.206 | | 0.800 | ***********....................*..............*****....*****X | 90.697 | | 8.699 | ..****X | 8.699 | | 3.700 | X | 3.700 | | 0.800 | *****X | 0.900 | | 0.400 | ***X | 0.400 | | 0.300 | *****X | 1.500 | | 0.900 | ****X | 0.900 | | 0.800 | *****X | 1.000 | | 0.500 | **********X | 1.200 | | 0.700 | X | 0.700 | | 0.600 | ****X | 0.700 | | 0.400 | **********X | 1.900 | | 0.100 | *****...*....................*****..X | 60.001 | | 49.200 | ...................****X | 49.200 | | 36.401 | ..............*****.......*.....****X | 64.604 | | 27.499 | ..........X | 27.499 | | 1.800 | ***X | 1.800 | | 1.400 | X | 1.400 | | 0.300 | X | 0.300 | | 0.200 | *********X | 1.200 | | 0.800 | ****X | 0.800 | | 0.700 | ************.****X | 3.400 | | 3.000 | X | 3.000 | | 0.800 | ***************X | 2.300 | | 2.100 | X | 2.100 | | 1.800 | ****...*.............*****.*.****X | 42.601 | | 41.101 | ................*....****X | 49.992 | | 45.792 | ..................X | 45.792 | | 34.401 | .............****X | 34.401 | | 1.000 | *****X | 1.000 | +---------+-----------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:08:28,398 | test | INFO | pool-26-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***............................*.........***X | 273.530 | | 139.770 | ..................X | 139.770 | | 111.157 | ..............***........................*.......................***........***X | 514.936 | | 356.400 | ...............................................X | 356.400 | | 159.122 | ....................****........................*............................*X | 535.846 | | 535.846 | ........................................................................*X | 536.068 | | 533.572 | .......................................................................**X | 533.572 | | 418.684 | ........................................................X | 418.684 | | 224.963 | .............................****..............*................***X | 440.131 | | 257.024 | ..................................****.........*.........***X | 388.229 | | 309.118 | .........................................X | 309.118 | | 227.821 | ..............................****.................*****.....*****......*..........****X | 505.274 | | 430.774 | .........................................................***X | 430.774 | | 430.774 | .........................................................X | 430.774 | | 364.182 | ................................................X | 364.182 | | 338.704 | .............................................****X | 338.704 | | 234.810 | ...............................******............****X | 323.614 | | 288.704 | ......................................X | 288.704 | | 256.608 | ..................................*****..*****..*.*****...............................................*X | 638.676 | | 638.676 | ......................................................................................**X | 638.676 | | 531.256 | .......................................................................X | 531.256 | | 360.530 | ................................................******..*****.X | 384.191 | | 365.807 | .................................................***X | 365.807 | | 341.809 | .............................................*.....*****X | 374.599 | | 355.106 | ...............................................X | 355.106 | | 355.106 | ...............................................***X | 355.106 | | 306.854 | ........................................*****.................*.........****X | 492.143 | | 369.215 | .................................................****X | 369.215 | | 359.723 | ................................................*.....*****......................*****..*.*****..................*****X | 710.348 | | 544.814 | .........................................................................****X | 544.814 | | 483.687 | .................................................................X | 483.687 | | 415.672 | .......................................................*****.....................****X | 563.502 | | 541.547 | .........................................................................X | 541.547 | | 498.015 | ...................................................................**X | 498.015 | | 498.015 | ...................................................................*X | 498.015 | | 449.841 | ............................................................*****.........*........****X | 575.099 | | 446.038 | ............................................................*****X | 446.038 | | 344.495 | ..............................................*****.....*.........****..........*............X | 608.644 | | 608.644 | ..................................................................................*X | 608.644 | | 608.644 | ..................................................................................*X | 608.644 | | 559.947 | ...........................................................................****X | 559.947 | | 507.740 | ....................................................................X | 507.740 | | 467.233 | ..............................................................****.*.*****X | 479.978 | | 405.433 | ......................................................****X | 405.433 | | 404.779 | ......................................................*.***X | 409.471 | | 407.563 | ......................................................X | 407.563 | | 324.537 | ...........................................****X | 324.537 | | 307.529 | .........................................X | 307.529 | | 273.981 | ....................................****X | 273.981 | | 243.187 | ................................X | 243.187 | | 228.992 | ..............................*****.............*****....*........****X | 411.945 | | 389.499 | ....................................................X | 389.499 | | 345.600 | ..............................................*****..........*****.....*.****..*.*****.*...*...***X | 537.379 | | 496.707 | ..................................................................X | 496.707 | | 471.304 | ...............................................................***X | 471.304 | | 470.485 | ...............................................................X | 470.485 | | 470.200 | ...............................................................******..........****X | 547.465 | | 491.033 | ..................................................................X | 491.033 | | 464.183 | ..............................................................****..*X | 480.703 | | 469.602 | ...............................................................***X | 469.602 | | 418.503 | ........................................................X | 418.503 | | 325.420 | ...........................................*****..*....*****......X | 414.807 | | 390.417 | ....................................................***X | 390.417 | | 331.424 | ............................................X | 331.424 | | 318.616 | ..........................................****..*.*****....*.....*****.......****X | 455.374 | | 437.682 | ..........................................................X | 437.682 | | 428.799 | .........................................................*****.X | 438.166 | | 413.859 | .......................................................****...*............****X | 521.399 | | 494.764 | ..................................................................X | 494.764 | | 393.634 | ....................................................******..............****X | 490.007 | | 470.322 | ...............................................................******.....*****..X | 523.425 | | 515.996 | .....................................................................***X | 515.996 | | 466.837 | ..............................................................X | 466.837 | | 319.509 | ..........................................*****....*..*****.*....****......*...........*****.X | 528.033 | | 511.200 | ....................................................................*****...............................X | 731.083 | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:08:29,592 | test | INFO | pool-26-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***...........*.......****.............*...**.............*.....*...............**X | 183.153 | | 114.336 | .........................................*X | 114.336 | | 114.336 | .........................................**.........*..................***X | 188.705 | | 139.084 | ..................................................X | 139.084 | | 92.088 | .................................***..................*..............**X | 180.804 | | 174.680 | ...............................................................X | 174.680 | | 126.542 | .............................................****............****.................X | 205.247 | | 149.069 | ......................................................****.................*..........................****..X | 270.741 | | 245.724 | .........................................................................................***X | 245.724 | | 202.194 | .........................................................................*****........................X | 267.948 | | 262.529 | ...............................................................................................****X | 262.529 | | 234.444 | .....................................................................................*.***X | 236.400 | | 196.617 | .......................................................................X | 196.617 | | 159.309 | .........................................................*****...*****X | 165.695 | | 153.502 | .......................................................*****..................*...............****X | 243.404 | | 176.610 | ................................................................X | 176.610 | | 149.720 | ......................................................*****........*...............***X | 212.101 | | 209.117 | ............................................................................*.......****X | 227.686 | | 133.820 | ................................................****X | 133.820 | | 91.314 | ................................*.****X | 94.406 | | 2.401 | ****X | 2.401 | | 0.900 | ***********X | 1.400 | | 1.100 | X | 1.100 | | 0.600 | ***X | 0.600 | | 0.000 | **********X | 0.800 | | 0.700 | X | 0.700 | | 0.600 | ****X | 0.600 | | 0.000 | ****************X | 1.300 | | 0.700 | ************.............................****X | 79.472 | | 44.070 | ...............X | 44.070 | | 22.906 | .......*****.............****X | 57.189 | | 53.997 | ..................X | 53.997 | | 39.004 | .............****X | 39.004 | | 31.907 | ..........X | 31.907 | | 0.200 | *****...........................****X | 74.990 | | 74.182 | ..........................*..****X | 81.023 | | 50.323 | .................X | 50.323 | | 6.701 | .****X | 6.701 | | 0.500 | ****X | 0.500 | | 0.500 | *****X | 0.600 | | 0.100 | **********X | 2.301 | | 1.400 | X | 1.400 | | 0.600 | **********....*..........*..........***X | 65.207 | | 61.106 | .....................X | 61.106 | | 31.598 | ..........*****.*..*****.........*.....****X | 77.891 | | 74.392 | ..........................X | 74.392 | | 63.308 | ......................***X | 63.308 | | 62.603 | ......................*....*****X | 75.800 | | 21.599 | ......****X | 21.599 | | 0.800 | ****X | 0.800 | | 0.400 | *****X | 0.400 | | 0.000 | ***********X | 1.000 | | 1.000 | ***X | 1.000 | | 0.800 | X | 0.800 | | 0.600 | **********X | 1.200 | | 0.000 | ****************X | 2.100 | | 1.000 | X | 1.000 | | 0.900 | ***X | 0.900 | | 0.800 | X | 0.800 | | 0.600 | X | 0.600 | | 0.600 | ****...*................****X | 52.599 | | 47.498 | ................*****X | 47.602 | | 0.900 | *****.........*......****X | 42.100 | | 39.396 | .............*****X | 39.396 | | 14.099 | ....X | 14.099 | | 0.000 | ********X | 1.700 | | 1.300 | X | 1.300 | | 1.200 | X | 1.200 | | 1.100 | ***X | 1.100 | | 0.400 | ***********...*.......****...*......*****X | 53.201 | | 52.001 | ..................****X | 52.001 | | 50.901 | .................X | 50.901 | | 49.491 | .................***X | 49.491 | | 37.992 | .............X | 37.992 | | 2.900 | *****.........*................****X | 71.500 | | 32.800 | ...........X | 32.800 | | 10.899 | ...X | 10.899 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:08:29,641 | test | WARNING | Thread-85108 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:Gwg3Y4iQ-10-818000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 11:08:29,690 | test | WARNING | Thread-85258 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:cq98M-nJI6Cf8nV4-10-846000 update failed keys from task: [u'test_collections-1059', u'test_collections-1022', u'test_collections-1045'] 2021-05-17 11:08:29,694 | test | WARNING | Thread-85270 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:kCQV-v9mgbm-10-847000 update failed keys from task: [u'test_collections-1016', u'test_collections-1068'] 2021-05-17 11:08:29,697 | test | WARNING | Thread-85285 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Sh-10-856000:Clg--10-856000 update failed keys from task: [u'test_collections-1190', u'test_collections-1137', u'test_collections-1172', u'test_collections-1074', u'test_collections-1230', u'test_collections-1252', u'test_collections-1155'] 2021-05-17 11:08:29,701 | test | WARNING | Thread-85298 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Sh-10-856000:vMDyuWfbAKeqYydl-10-857000 update failed keys from task: [u'test_collections-1093', u'test_collections-988', u'test_collections-1103', u'test_collections-1143', u'test_collections-1000', u'test_collections-1022', u'test_collections-1045', u'test_collections-1146'] 2021-05-17 11:08:29,703 | test | WARNING | Thread-85307 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Sh-10-856000:voncfT9jCflszxkM4FS-10-857000 update failed keys from task: [u'test_collections-1270', u'test_collections-982', u'test_collections-1235', u'test_collections-1423', u'test_collections-1325', u'test_collections-1339', u'test_collections-1429', u'test_collections-1266', u'test_collections-1223'] 2021-05-17 11:08:29,709 | test | WARNING | Thread-85321 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:af1F9fO-Yko1IiTk-2-980000:E-3-114000 update failed keys from task: [u'test_collections-1926', u'test_collections-1901', u'test_collections-1891'] 2021-05-17 11:08:29,723 | test | WARNING | Thread-85251 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:5TA95GMJEJfGmk8N_-10-846000 update failed keys from task: [u'test_collections-1093', u'test_collections-1103', u'test_collections-1186', u'test_collections-1143', u'test_collections-1045', u'test_collections-1146'] 2021-05-17 11:08:29,726 | test | WARNING | Thread-85181 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 update failed keys from task: [u'test_collections-1706', u'test_collections-1743', u'test_collections-1622', u'test_collections-1820', u'test_collections-1667', u'test_collections-1865', u'test_collections-1668', u'test_collections-1662', u'test_collections-1738', u'test_collections-1653', u'test_collections-1930', u'test_collections-1737', u'test_collections-1772', u'test_collections-1696', u'test_collections-1848', u'test_collections-1929', u'test_collections-1764', u'test_collections-1600', u'test_collections-1645', u'test_collections-1887', u'test_collections-1724', u'test_collections-1761', u'test_collections-1917', u'test_collections-1952', u'test_collections-1833', u'test_collections-1790', u'test_collections-1750'] 2021-05-17 11:08:29,778 | test | WARNING | Thread-85492 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 update failed keys from task: [u'test_collections-1270', u'test_collections-1507', u'test_collections-1444', u'test_collections-1085', u'test_collections-1560', u'test_collections-1266', u'test_collections-1223', u'test_collections-1080', u'test_collections-1235', u'test_collections-1115', u'test_collections-1339', u'test_collections-1539', u'test_collections-1110', u'test_collections-1452', u'test_collections-1497'] 2021-05-17 11:08:29,778 | test | WARNING | Thread-85496 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HuCELxXpvN4amFSUWI-6-316000:S-VkqImDKX-6-474000 create failed keys from task: [u'test_collections-570'] 2021-05-17 11:08:29,785 | test | WARNING | Thread-85321 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:af1F9fO-Yko1IiTk-2-980000:E-3-114000 delete failed keys from task: [u'test_collections-713', u'test_collections-535', u'test_collections-734', u'test_collections-566', u'test_collections-479', u'test_collections-465', u'test_collections-672', u'test_collections-683', u'test_collections-563', u'test_collections-541'] 2021-05-17 11:08:29,786 | test | WARNING | Thread-85498 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:X_D_RYQRe7bk-7-1000:b-iF5QGritfbmL2Vs-7-136000 create failed keys from task: [u'test_collections-526'] 2021-05-17 11:08:29,848 | test | WARNING | Thread-85342 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:EkQo-10-853000:il9spZ--10-855000 update failed keys from task: [u'test_collections-988', u'test_collections-1022', u'test_collections-1045'] 2021-05-17 11:08:29,858 | test | WARNING | Thread-85401 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:-j5VS-10-853000 update failed keys from task: [u'test_collections-1270', u'test_collections-982', u'test_collections-1423', u'test_collections-1325', u'test_collections-1429', u'test_collections-1183', u'test_collections-1382', u'test_collections-1266', u'test_collections-1223', u'test_collections-1235', u'test_collections-1150', u'test_collections-1074', u'test_collections-1394'] 2021-05-17 11:08:29,862 | test | WARNING | Thread-85408 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:te2XeiST-10-852000 delete failed keys from task: [u'test_collections-933', u'test_collections-973'] 2021-05-17 11:08:30,332 | test | WARNING | Thread-85492 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dChuMvbyS-1-166000:171MjdarUkv-1-344000 delete failed keys from task: [u'test_collections-425', u'test_collections-335', u'test_collections-552', u'test_collections-460'] 2021-05-17 11:08:30,723 | test | WARNING | Thread-85181 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:98yBWaeEpc_mf-0-367000:upz_opGBy692C-0-517000 delete failed keys from task: [u'test_collections-947', u'test_collections-925', u'test_collections-756', u'test_collections-960'] 2021-05-17 11:08:43,960 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 11:10:02,403 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 11:10:02,403 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 11:10:20,480 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 11:10:20,480 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 11:10:34,246 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 11:10:34,246 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 11:11:33,341 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1339998245 seconds to complete 2021-05-17 11:11:33,355 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3285.467s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 24 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_24 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_25 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_25', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 25, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 11:11:42,908 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #25 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 11:11:43,625 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #25 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 11:11:44,063 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe47da9 type=client open_count=1 channel=[id: 0xb3bd7d79, 0.0.0.0/0.0.0.0:53934] timeout=300.0>) 2021-05-17 11:12:04,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe47daa type=client open_count=1 channel=[id: 0xa35a66a6, 0.0.0.0/0.0.0.0:35420] timeout=300.0>) 2021-05-17 11:12:06,934 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe47dab type=client open_count=1 channel=[id: 0xb43e392d, 0.0.0.0/0.0.0.0:55936] timeout=300.0>) 2021-05-17 11:12:08,956 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xe47dac type=client open_count=1 channel=[id: 0xb54354f1, 0.0.0.0/0.0.0.0:48888] timeout=300.0>) 2021-05-17 11:12:33,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xe47dad type=client open_count=1 channel=[id: 0xa1863974, 0.0.0.0/0.0.0.0:57222] timeout=300.0>) 2021-05-17 11:12:35,326 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xe47dae type=client open_count=1 channel=[id: 0x3553f3ba, 0.0.0.0/0.0.0.0:51581] timeout=300.0>) 2021-05-17 11:12:37,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xe47daf type=client open_count=1 channel=[id: 0xdacdf0ec, 0.0.0.0/0.0.0.0:47932] timeout=300.0>) 2021-05-17 11:13:01,249 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xe47db0 type=client open_count=1 channel=[id: 0xfbd14574, 0.0.0.0/0.0.0.0:46528] timeout=300.0>) 2021-05-17 11:13:03,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xe47db1 type=client open_count=1 channel=[id: 0xa7ed9557, 0.0.0.0/0.0.0.0:52101] timeout=300.0>) 2021-05-17 11:13:05,283 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xe47db2 type=client open_count=1 channel=[id: 0xb003c87b, 0.0.0.0/0.0.0.0:59632] timeout=300.0>) 2021-05-17 11:13:29,348 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xe47db3 type=client open_count=1 channel=[id: 0x14890a9f, 0.0.0.0/0.0.0.0:53983] timeout=300.0>) 2021-05-17 11:13:31,367 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xe47db4 type=client open_count=1 channel=[id: 0x76345528, 0.0.0.0/0.0.0.0:58754] timeout=300.0>) 2021-05-17 11:13:33,404 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xe47db5 type=client open_count=1 channel=[id: 0x4acb38ef, 0.0.0.0/0.0.0.0:52169] timeout=300.0>) 2021-05-17 11:13:57,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xe47db6 type=client open_count=1 channel=[id: 0xd92ef92d, 0.0.0.0/0.0.0.0:58940] timeout=300.0>) 2021-05-17 11:13:59,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xe47db7 type=client open_count=1 channel=[id: 0x4201748b, 0.0.0.0/0.0.0.0:42228] timeout=300.0>) 2021-05-17 11:14:01,279 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 11:14:18,736 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 11:14:32,443 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 11:14:32,464 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #25 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 11:14:32,500 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #25 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 11:14:45,832 | test | INFO | pool-27-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.155002583376 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 11:14:45,858 | test | INFO | pool-27-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 11:14:55,930 | test | INFO | pool-27-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 11:14:55,957 | test | INFO | pool-27-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-17 11:14:56,206 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 11:15:43,147 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 11:15:43,401 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 657045096 | 92821550 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 690853712 | 108690164 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 887845240 | 64952314 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 11:15:43,403 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 11:16:30,457 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 11:16:30,466 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 11:33:41,933 | test | INFO | pool-27-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***................................................*..................***X | 13.495 | | 9.895 | ...............................................X | 9.895 | | 0.000 | **************************************.....*..***X | 1.499 | | 0.700 | ..X | 0.700 | | 0.000 | ***********............*....*****....................................................................................*****X | 20.301 | | 9.299 | ............................................X | 9.299 | | 4.900 | .......................****...*..**********X | 6.000 | | 1.600 | ......X | 1.600 | | 0.000 | ***************************.......*****.....*..****X | 3.000 | | 2.699 | ............X | 2.699 | | 1.500 | ......*****.............................*****X | 7.501 | | 0.000 | ****************************************......................****X | 4.500 | | 4.099 | ...................X | 4.099 | | 3.000 | .............****X | 3.000 | | 1.600 | ......X | 1.600 | | 0.000 | *****......................*****X | 4.499 | | 0.000 | ****************.*....*..***X | 1.501 | | 0.000 | ***********.*......*****...........................................*****X | 10.208 | | 4.399 | ....................X | 4.399 | | 1.800 | .......****.....*.****X | 3.000 | | 0.000 | ***********************..*.....*****X | 1.501 | | 1.501 | ......****X | 1.501 | | 0.000 | *****.............*................****X | 5.902 | | 0.100 | *****X | 0.100 | | 0.000 | *****.................*.....................................................*............................***X | 19.909 | | 13.711 | ..................................................................X | 13.711 | | 4.101 | ...................*****........................****X | 9.003 | | 7.503 | ...................................X | 7.503 | | 1.500 | ......*********X | 1.501 | | 1.501 | ......X | 1.501 | | 1.500 | ......****X | 1.500 | | 1.200 | ....X | 1.200 | | 0.000 | *****..............******........******..................................................****X | 14.704 | | 13.204 | ................................................................X | 13.204 | | 10.802 | ....................................................****X | 10.802 | | 0.000 | ****************.....*............*****..X | 4.002 | | 2.400 | ..........*****........................................****X | 10.400 | | 9.200 | ............................................X | 9.200 | | 4.601 | .....................****X | 4.601 | | 2.301 | ..........X | 2.301 | | 1.500 | ......****X | 1.500 | | 0.000 | *****.....................*......................................*......................***X | 16.506 | | 14.105 | ....................................................................X | 14.105 | | 4.499 | .....................****X | 4.499 | | 3.001 | .............*****.....................*........................****X | 12.001 | | 3.203 | ..............X | 3.203 | | 0.000 | *****......*........***X | 3.001 | | 2.701 | ............X | 2.701 | | 1.500 | ......****X | 1.500 | | 0.000 | *****....................*........................****X | 9.000 | | 5.700 | ...........................X | 5.700 | | 4.501 | .....................****X | 4.501 | | 0.000 | ******..................................................................****X | 13.402 | | 10.402 | ..................................................X | 10.402 | | 8.002 | ......................................X | 8.002 | | 1.600 | ......****..........*............****X | 6.001 | | 5.301 | .........................X | 5.301 | | 4.501 | .....................****X | 4.501 | | 3.101 | ..............X | 3.101 | | 1.500 | ......***X | 1.500 | | 1.400 | .....X | 1.400 | | 0.200 | *****.......*......****X | 2.801 | | 0.000 | ******......................****X | 4.499 | | 3.800 | .................X | 3.800 | | 3.100 | ..............X | 3.100 | | 2.701 | ............****......*...........****X | 6.101 | | 3.101 | ..............X | 3.101 | | 1.700 | .......****X | 1.700 | | 0.300 | X | 0.300 | | 0.000 | ******************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 11:33:43,006 | test | INFO | pool-27-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..........................................*............****.....................*.........................****X | 799.049 | | 571.023 | ......................................................................*****.*.....****X | 618.929 | | 608.704 | ...........................................................................X | 608.704 | | 601.776 | ..........................................................................****...*............****X | 726.305 | | 677.462 | ...................................................................................****X | 677.462 | | 652.273 | ................................................................................X | 652.273 | | 622.122 | ............................................................................***X | 622.122 | | 516.392 | ...............................................................X | 516.392 | | 433.434 | .....................................................*****.....****X | 473.176 | | 458.058 | ........................................................*.X | 467.855 | | 448.772 | .......................................................***X | 448.772 | | 423.800 | ....................................................*.****X | 436.014 | | 369.775 | .............................................X | 369.775 | | 341.422 | .........................................*****...................*****X | 493.180 | | 472.657 | ..........................................................*****.X | 481.292 | | 460.956 | ........................................................X | 460.956 | | 453.068 | .......................................................***X | 453.068 | | 425.791 | ....................................................X | 425.791 | | 414.137 | ..................................................*****.*..........****.....*****X | 540.737 | | 491.908 | ............................................................******....****X | 526.793 | | 496.380 | .............................................................X | 496.380 | | 489.709 | ............................................................****X | 489.709 | | 466.300 | .........................................................X | 466.300 | | 463.480 | .........................................................*****.....****X | 506.181 | | 497.510 | .............................................................X | 497.510 | | 486.456 | ...........................................................****X | 486.456 | | 452.328 | .......................................................X | 452.328 | | 444.196 | ......................................................*****......****X | 494.019 | | 481.051 | ...........................................................*.****X | 488.647 | | 448.321 | .......................................................X | 448.321 | | 420.660 | ...................................................*****............******........*****X | 582.973 | | 514.564 | ...............................................................X | 514.564 | | 456.741 | ........................................................***X | 456.741 | | 456.341 | ........................................................*.*****.*.*****......****X | 534.993 | | 475.347 | ..........................................................X | 475.347 | | 416.638 | ...................................................*****.......*****X | 477.893 | | 458.876 | ........................................................X | 458.876 | | 439.535 | ......................................................****..*....****X | 494.968 | | 476.130 | ..........................................................X | 476.130 | | 461.663 | ........................................................***X | 461.663 | | 451.773 | .......................................................X | 451.773 | | 451.224 | .......................................................*****.........****X | 525.819 | | 499.978 | .............................................................X | 499.978 | | 454.749 | .......................................................****X | 454.749 | | 433.484 | .....................................................X | 433.484 | | 417.696 | ...................................................*****.........*****X | 491.656 | | 392.861 | ................................................****X | 392.861 | | 372.385 | .............................................X | 372.385 | | 335.902 | .........................................*****....*................****X | 496.315 | | 483.588 | ...........................................................*****X | 483.588 | | 412.221 | ..................................................******..******..*****...........*....****X | 560.925 | | 444.160 | ......................................................*****.*....******.****X | 488.755 | | 399.278 | ................................................*****..*........****X | 479.045 | | 430.208 | ....................................................X | 430.208 | | 412.261 | ..................................................*****......******..*****..*.*****......****X | 546.745 | | 441.123 | ......................................................X | 441.123 | | 388.134 | ...............................................*****...*.............****X | 514.668 | | 387.796 | ...............................................*****...*.......**********..*....****X | 519.230 | | 512.030 | ...............................................................X | 512.030 | | 477.185 | ..........................................................****.*...****X | 509.495 | | 509.200 | ..............................................................X | 509.200 | | 495.156 | ............................................................****X | 495.156 | | 440.134 | ......................................................X | 440.134 | | 410.713 | ..................................................*****...*.****.....*....****X | 514.828 | | 502.908 | .............................................................X | 502.908 | | 462.764 | ........................................................****X | 462.764 | | 368.040 | .............................................*****.X | 382.622 | | 360.356 | ............................................*****...............****X | 487.016 | | 485.332 | ...........................................................X | 485.332 | | 460.957 | ........................................................****X | 460.957 | | 455.037 | .......................................................*..*****..........****X | 550.391 | | 462.030 | ........................................................X | 462.030 | | 428.417 | ....................................................*****...*........****X | 513.396 | | 422.034 | ...................................................*****..*.........*****.X | 515.710 | | 500.545 | .............................................................****X | 500.545 | | 460.223 | ........................................................X | 460.223 | | 445.602 | ......................................................*****...******...*****.X | 502.094 | | 499.194 | .............................................................****X | 501.558 | | 448.215 | .......................................................X | 448.215 | | 422.599 | ...................................................*****............*...****X | 542.282 | | 494.562 | ............................................................X | 494.562 | | 476.789 | ..........................................................***X | 476.789 | | 476.751 | ..........................................................X | 476.751 | | 454.487 | .......................................................*****....*...*****..*****X | 520.653 | | 493.207 | ............................................................****X | 493.207 | | 483.607 | ...........................................................*....****X | 518.691 | | 401.843 | .................................................X | 401.843 | | 347.632 | ..........................................*****..............*****......*...*....****...*..**X | 606.164 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:33:43,849 | test | INFO | pool-27-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***...........*.....***X | 2.999 | | 1.399 | ......X | 1.399 | | 0.000 | *******************************************************......*..................*****........*........****X | 7.500 | | 3.100 | ...............X | 3.100 | | 1.500 | .......*****....*............*****........................*****X | 8.999 | | 1.500 | .......******......*..***X | 2.999 | | 0.000 | ****************.*................*.......****..................*......*****..****X | 9.397 | | 8.997 | ...............................................X | 8.997 | | 4.999 | ..........................X | 4.999 | | 2.600 | .............***X | 2.600 | | 0.400 | .X | 0.400 | | 0.000 | *****..*......****X | 1.500 | | 0.000 | ******.................................................****X | 9.078 | | 4.201 | .....................X | 4.201 | | 0.000 | ***********...............*****...........*........*......................***X | 10.499 | | 8.699 | ..............................................X | 8.699 | | 1.500 | .......****X | 1.500 | | 0.000 | *****........*........................****X | 5.999 | | 4.799 | ........................X | 4.799 | | 0.000 | ***********........*****............*....****X | 4.504 | | 0.000 | ***********..........*.............................*****..****X | 7.605 | | 7.005 | ....................................X | 7.005 | | 4.603 | .......................****X | 4.603 | | 4.301 | ......................X | 4.301 | | 3.000 | ...............****X | 3.000 | | 0.000 | *****************...*.............****X | 3.001 | | 0.000 | ******........................*****X | 4.503 | | 0.000 | *****................*****X | 3.001 | | 1.801 | ........X | 1.801 | | 0.000 | **********.........................*.......................................****X | 12.005 | | 5.401 | ............................X | 5.401 | | 3.001 | ...............*****........****X | 4.501 | | 2.402 | ...........X | 2.402 | | 0.000 | *****...........*.............***X | 4.502 | | 3.602 | ..................X | 3.602 | | 0.000 | **********....*............****X | 3.001 | | 1.801 | ........X | 1.801 | | 0.600 | ..*****..........................******...........................................................****X | 16.403 | | 13.103 | .....................................................................X | 13.103 | | 0.000 | **********************.......................................******..****X | 7.600 | | 6.802 | ...................................X | 6.802 | | 6.002 | ...............................****X | 6.002 | | 3.901 | ....................X | 3.901 | | 1.500 | .......****.*..........*****.................................................................................****X | 18.506 | | 14.803 | ..............................................................................X | 14.803 | | 10.798 | .........................................................****X | 10.798 | | 5.003 | ..........................X | 5.003 | | 3.001 | ...............*****................*****X | 6.000 | | 3.201 | ................X | 3.201 | | 0.000 | **********....*....................****X | 4.501 | | 0.000 | *****.....................................*................................................****X | 15.800 | | 7.300 | ......................................X | 7.300 | | 3.701 | ..................****X | 3.701 | | 0.000 | *****........*..........................****X | 6.301 | | 5.701 | .............................X | 5.701 | | 2.700 | .............****X | 2.700 | | 2.400 | ...........X | 2.400 | | 1.500 | .......****X | 1.500 | | 1.400 | ......******..................................................................................****X | 16.601 | | 13.301 | ......................................................................X | 13.301 | | 0.000 | **********************************................*****X | 3.000 | | 0.000 | ********************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 11:33:44,127 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 80.8343611454 | 6437847040 | 4291911680 | 2451263488 / 7511588864 | 101472 / 202735 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 87.2412413747 | 6437847040 | 4244922368 | 2452127744 / 7511588864 | 99512 / 201557 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 67.2644544076 | 6437847040 | 4228476928 | 2486095872 / 7511588864 | 100805 / 201923 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 60.0006666556 | 6437847040 | 3806998528 | 3084718080 / 9530753024 | 100493 / 197373 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 61.7183333333 | 6437847040 | 4241162240 | 2477318144 / 7511588864 | 101100 / 203182 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 11:33:57,125 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 11:35:23,601 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164418776 | 261090652 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162147392 | 277927694 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 444659608 | 465045377 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 11:35:23,602 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #25 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 11:35:33,762 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 11:35:33,763 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 11:35:33,763 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 11:36:15,742 | test | INFO | pool-27-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 41.7939999104 seconds to complete 2021-05-17 11:36:15,746 | test | INFO | pool-27-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 11:38:20,805 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 11:38:20,805 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 11:38:20,806 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 11:38:28,496 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 11:50:37,017 | test | INFO | pool-27-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..*....*****................*.......*****...................................................*****X | 204.000 | | 184.303 | .......................................................................X | 184.303 | | 160.928 | ..............................................................*****...*****X | 169.409 | | 151.395 | ..........................................................X | 151.395 | | 139.267 | .....................................................*****....******.............*...****......X | 204.579 | | 200.388 | ..............................................................................*****........*.............****X | 253.566 | | 207.826 | ................................................................................****X | 207.826 | | 205.136 | ...............................................................................X | 205.136 | | 198.415 | .............................................................................****X | 198.415 | | 193.611 | ...........................................................................X | 193.611 | | 187.886 | .........................................................................*****......X | 203.380 | | 199.094 | .............................................................................****X | 199.094 | | 157.497 | .............................................................X | 157.497 | | 151.696 | ..........................................................****.....................*.......*****.X | 224.984 | | 221.273 | ......................................................................................****X | 221.273 | | 140.508 | ......................................................X | 140.508 | | 65.305 | ........................**X | 65.305 | | 65.305 | ........................*X | 65.305 | | 45.800 | .................X | 45.800 | | 35.099 | ............*****.....******...***********X | 55.195 | | 46.301 | .................*****X | 46.301 | | 45.805 | .................******...*****X | 55.093 | | 49.709 | ..................*****X | 49.709 | | 49.312 | ..................*..****X | 54.403 | | 48.778 | ..................******..*****X | 54.904 | | 1.600 | *****X | 1.600 | | 1.500 | *****X | 1.500 | | 1.000 | *****.**X | 4.801 | | 4.801 | **X | 4.801 | | 1.397 | *****X | 1.397 | | 1.097 | ************.****X | 4.393 | | 3.293 | X | 3.293 | | 1.297 | *****.*****X | 4.493 | | 1.897 | *****X | 1.897 | | 0.899 | ***********X | 1.398 | | 1.198 | X | 1.198 | | 0.599 | ***********.**X | 3.396 | | 3.396 | **X | 3.396 | | 2.796 | X | 2.796 | | 2.198 | *****.******.****X | 5.993 | | 5.693 | .X | 5.693 | | 4.495 | ****X | 4.495 | | 2.695 | X | 2.695 | | 1.199 | *****************X | 2.497 | | 2.398 | *****.*.*****X | 5.494 | | 5.296 | .*****X | 5.296 | | 1.698 | ****X | 1.698 | | 1.498 | X | 1.498 | | 1.199 | *****X | 1.199 | | 0.000 | ******.*****X | 4.196 | | 1.299 | ***********X | 1.999 | | 1.799 | X | 1.799 | | 1.399 | *****.*****X | 4.596 | | 2.200 | ******.***********X | 3.698 | | 3.198 | X | 3.198 | | 2.099 | **********X | 2.099 | | 0.900 | *******.***X | 2.898 | | 2.698 | X | 2.698 | | 2.599 | ****X | 2.599 | | 2.399 | *****X | 2.399 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:50:37,065 | test | INFO | pool-27-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***............*..........................................****X | 407.431 | | 379.900 | ..................................................******...................******..............................****X | 743.630 | | 609.038 | ................................................................................X | 609.038 | | 577.832 | ............................................................................*X | 577.832 | | 577.832 | ............................................................................**X | 577.832 | | 511.587 | ...................................................................X | 511.587 | | 462.887 | .............................................................****X | 462.887 | | 322.782 | ..........................................*****X | 322.782 | | 202.499 | ..........................*****..............*...............*****.*.****X | 433.438 | | 361.657 | ...............................................X | 361.657 | | 330.340 | ...........................................****X | 330.340 | | 307.098 | ........................................******..X | 321.600 | | 320.274 | ..........................................***X | 320.274 | | 320.274 | ..........................................*......*..****X | 386.192 | | 278.899 | ....................................X | 278.899 | | 235.094 | ..............................***X | 235.094 | | 231.100 | ..............................X | 231.100 | | 221.105 | ............................****X | 221.105 | | 207.079 | ..........................*.******...........................*****X | 410.219 | | 348.100 | .............................................******................****X | 466.041 | | 412.057 | ......................................................X | 412.057 | | 183.592 | .......................*****....*................****X | 334.465 | | 314.159 | .........................................X | 314.159 | | 229.406 | .............................*****..*...****X | 265.626 | | 265.419 | ..................................*....*****....******.................*.......****X | 498.325 | | 378.500 | .................................................X | 378.500 | | 357.887 | ...............................................*****.X | 366.304 | | 350.226 | ..............................................****X | 350.226 | | 333.625 | ...........................................******.............................*X | 543.627 | | 543.627 | ........................................................................*X | 543.627 | | 543.627 | ........................................................................*X | 543.627 | | 447.771 | ...........................................................****X | 447.771 | | 372.540 | .................................................X | 372.540 | | 281.456 | ....................................*****........*...****X | 360.740 | | 301.999 | .......................................******.............*X | 394.508 | | 394.508 | ....................................................****....................*****X | 546.732 | | 436.072 | .........................................................*****X | 436.072 | | 318.630 | .........................................*****X | 318.630 | | 294.309 | ......................................X | 294.309 | | 251.439 | ................................*****.........................X | 435.125 | | 435.125 | .........................................................****X | 435.125 | | 409.970 | ......................................................X | 409.970 | | 391.466 | ...................................................***X | 391.466 | | 337.256 | ............................................X | 337.256 | | 296.561 | ......................................*****.....X | 329.721 | | 329.144 | ...........................................*****................................*X | 566.374 | | 566.374 | ...........................................................................****......*****X | 610.781 | | 607.352 | ................................................................................*****X | 607.352 | | 516.163 | ....................................................................****X | 516.163 | | 493.328 | .................................................................X | 493.328 | | 484.166 | ................................................................****X | 484.166 | | 432.899 | .........................................................******.....******...*****X | 494.003 | | 397.668 | ....................................................******..*.........****X | 479.309 | | 447.798 | ...........................................................X | 447.798 | | 413.987 | ......................................................*****.....*.****X | 454.058 | | 450.701 | ...........................................................*..****X | 463.919 | | 439.517 | ..........................................................X | 439.517 | | 425.598 | ........................................................*****..X | 441.978 | | 441.778 | ..........................................................*****.X | 451.056 | | 436.752 | .........................................................****X | 436.752 | | 436.518 | .........................................................*.....****X | 474.100 | | 449.768 | ...........................................................X | 449.768 | | 416.231 | ......................................................*****......*...****X | 481.647 | | 425.438 | ........................................................X | 425.438 | | 386.861 | ...................................................*****.......*.....*****...*...****X | 525.338 | | 514.838 | ....................................................................*..*****.X | 535.966 | | 491.242 | .................................................................**********.....**..****.........*..*****..*****X | 642.118 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:50:37,757 | test | INFO | pool-27-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********X | 0.800 | | 0.000 | ******.............******.....................*****X | 56.310 | | 43.202 | .........................****X | 43.202 | | 42.398 | ........................*.*****....*.****X | 50.713 | | 48.806 | ............................*.****X | 49.200 | | 49.001 | .............................X | 49.001 | | 48.299 | ............................*****.*..****X | 53.493 | | 47.809 | ............................*.*****..*X | 53.104 | | 53.104 | ...............................**X | 53.104 | | 49.703 | .............................X | 49.703 | | 42.397 | ........................*****......*..*****...*.***X | 60.998 | | 50.800 | ..............................X | 50.800 | | 47.101 | ...........................*****......*.......****X | 68.091 | | 62.903 | .....................................X | 62.903 | | 59.203 | ...................................*****......*...*****..........****X | 91.389 | | 86.795 | ....................................................X | 86.795 | | 73.896 | ............................................*****...*.....****X | 87.192 | | 77.188 | ..............................................X | 77.188 | | 69.202 | .........................................*****....X | 76.406 | | 75.808 | .............................................*****....*........****X | 95.186 | | 83.184 | .................................................X | 83.184 | | 78.117 | ..............................................*****......*............****X | 106.405 | | 106.100 | ...............................................................*.....*****.*.......X | 126.639 | | 125.609 | ...........................................................................*****...******................****X | 155.880 | | 154.452 | .............................................................................................X | 154.452 | | 150.357 | ...........................................................................................****X | 150.357 | | 116.325 | ......................................................................******.............................*****X | 163.298 | | 154.690 | .............................................................................................X | 154.690 | | 138.778 | ...................................................................................****X | 138.778 | | 125.481 | ...........................................................................*.****......*...****X | 141.954 | | 134.648 | .................................................................................X | 134.648 | | 122.052 | .........................................................................*****............*****X | 140.986 | | 70.729 | ..........................................******.******..X | 75.830 | | 75.830 | .............................................*X | 75.830 | | 75.830 | .............................................**.X | 77.728 | | 74.747 | ............................................*****.*.****X | 77.941 | | 22.101 | ............X | 22.101 | | 6.196 | ..****X | 6.196 | | 3.795 | .******.******X | 5.293 | | 2.498 | **********X | 3.197 | | 1.998 | ******X | 2.398 | | 1.898 | **********X | 2.597 | | 2.098 | ******.******.*****X | 5.296 | | 2.498 | *****X | 2.498 | | 1.898 | *****X | 1.898 | | 0.799 | ******.*****X | 2.398 | | 2.198 | X | 2.198 | | 1.299 | *****.************.******X | 4.197 | | 3.499 | .*****X | 3.999 | | 3.898 | .****X | 3.898 | | 3.398 | .X | 3.398 | | 3.098 | *****.X | 3.498 | | 3.398 | .*****X | 3.398 | | 2.199 | ****X | 2.199 | | 2.099 | ************..****X | 5.198 | | 4.099 | .X | 4.099 | | 3.499 | .*****X | 3.499 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 11:50:38,032 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 11:50:38,084 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 11:50:48,352 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 11:50:48,513 | test | INFO | pool-27-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.390006500108 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 40.9903003233 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 34.0104827402 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 61.4018395179 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 49.6599886663 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 11:50:48,542 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 11:50:53,648 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 11:50:57,805 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 11:50:58,739 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.54 2021-05-17 11:51:03,816 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 3.09 2021-05-17 11:51:08,884 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 5.86 2021-05-17 11:51:11,017 | test | INFO | pool-27-thread-26 | [table_view:display:72] Keys failed in bucket1:_default:-gEf1vIqdx4dzA-56-273000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:51:13,795 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 11:51:13,983 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 9.11 2021-05-17 11:51:19,065 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 12.31 2021-05-17 11:51:24,153 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 16.64 2021-05-17 11:51:29,250 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 20.65 2021-05-17 11:51:34,328 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 24.66 2021-05-17 11:51:39,424 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 27.81 2021-05-17 11:51:44,515 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 30.73 2021-05-17 11:51:49,642 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 11:51:54,328 | test | INFO | pool-27-thread-10 | [table_view:display:72] Keys failed in bucket1:ToVxURDN%t3Wk6t-53-769000:ibYsI3-53-898000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-456 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:51:54,329 | test | INFO | pool-27-thread-10 | [table_view:display:72] Keys failed in bucket1:ToVxURDN%t3Wk6t-53-769000:ibYsI3-53-898000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-456 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:51:54,719 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 35.45 2021-05-17 11:51:59,576 | test | INFO | pool-27-thread-9 | [table_view:display:72] Keys failed in bucket2:1cbvs51B-24-617000:fiVH-24-769000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:51:59,576 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in bucket2:tXD-25-280000:782d661_URgFQ5n-28-290000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:51:59,789 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 38.92 2021-05-17 11:52:04,884 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 41.9 2021-05-17 11:52:09,973 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 45.04 2021-05-17 11:52:15,076 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 48.35 2021-05-17 11:52:17,355 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in bucket2:a0k-24-961000:pNIeqrGWM0Ondg-25-130000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1117 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1030 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1020 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1075 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1087 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1176 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1133 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1057 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:20,184 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 52.2 2021-05-17 11:52:25,269 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 55.56 2021-05-17 11:52:28,407 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in bucket2:a0k-24-961000:pNIeqrGWM0Ondg-25-130000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1390 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1281 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1378 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1235 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1354 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1200 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1311 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:30,357 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 58.92 2021-05-17 11:52:35,496 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 62.28 2021-05-17 11:52:39,982 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in bucket2:a0k-24-961000:pNIeqrGWM0Ondg-25-130000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1518 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1554 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1488 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1523 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1559 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1515 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1407 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1440 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1485 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1442 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1597 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:40,605 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 65.53 2021-05-17 11:52:45,686 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.88 2021-05-17 11:52:49,782 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in bucket2:a0k-24-961000:pNIeqrGWM0Ondg-25-130000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:49,993 | test | INFO | pool-27-thread-26 | [table_view:display:72] Keys failed in bucket2:eI8-53-338000:Ko-LKv9-53-451000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1718 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1688 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:49,993 | test | INFO | pool-27-thread-1 | [table_view:display:72] Keys failed in bucket2:PqiRsn8V1jAjlAGRgLHI-53-0:SHavScxByA-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1718 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1688 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:49,994 | test | INFO | pool-27-thread-26 | [table_view:display:72] Keys failed in bucket2:eI8-53-338000:Ko-LKv9-53-451000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1718 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1688 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:49,996 | test | INFO | pool-27-thread-1 | [table_view:display:72] Keys failed in bucket2:PqiRsn8V1jAjlAGRgLHI-53-0:SHavScxByA-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1718 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1688 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:50,772 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 69.65 2021-05-17 11:52:54,311 | test | INFO | pool-27-thread-24 | [table_view:display:72] Keys failed in default:_default:awlCa-56-289000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1411 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:52:55,872 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 72.14 2021-05-17 11:52:57,430 | test | INFO | pool-27-thread-18 | [table_view:display:72] Keys failed in default:_default:EgDE-56-288000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1363 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:00,953 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 74.69 2021-05-17 11:53:01,048 | test | INFO | pool-27-thread-15 | [table_view:display:72] Keys failed in default:scope2:WmJYX2vN-56-293000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-715 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-812 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-833 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-782 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-750 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:06,039 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 77.24 2021-05-17 11:53:07,201 | test | INFO | pool-27-thread-30 | [table_view:display:72] Keys failed in default:Jv-56-613000:FpSTty6PF-56-771000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:07,203 | test | INFO | pool-27-thread-7 | [table_view:display:72] Keys failed in default:scope2:A6MOecKi__pz-56-293000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1279 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1218 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:07,203 | test | INFO | pool-27-thread-30 | [table_view:display:72] Keys failed in default:Jv-56-613000:FpSTty6PF-56-771000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1536 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:07,497 | test | INFO | pool-27-thread-9 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-539 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-529 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-625 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-614 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-604 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-689 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-612 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-567 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-655 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-598 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-661 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:07,624 | test | INFO | pool-27-thread-13 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1071 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-993 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-994 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1020 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1010 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:09,769 | test | INFO | pool-27-thread-16 | [table_view:display:72] Keys failed in default:scope2:fkPvY9-56-293000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1279 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1227 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1239 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:10,242 | test | INFO | pool-27-thread-27 | [table_view:display:72] Keys failed in default:scope2:KnehItU3q2Cui-56-292000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:11,127 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 80.27 2021-05-17 11:53:12,542 | test | INFO | pool-27-thread-18 | [table_view:display:72] Keys failed in default:_default:EgDE-56-288000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:16,017 | test | INFO | pool-27-thread-3 | [table_view:display:72] Keys failed in default:_default:2_A9Wug-56-287000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1081 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1050 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1005 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-991 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1117 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1062 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1152 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1075 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1087 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1111 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:16,227 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 82.82 2021-05-17 11:53:17,013 | test | INFO | pool-27-thread-15 | [table_view:display:72] Keys failed in default:scope2:WmJYX2vN-56-293000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-914 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-969 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-977 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:21,098 | test | INFO | pool-27-thread-7 | [table_view:display:72] Keys failed in default:scope2:A6MOecKi__pz-56-293000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1402 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1392 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:21,365 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 85.75 2021-05-17 11:53:24,065 | test | INFO | pool-27-thread-13 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1281 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1202 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1224 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1225 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1335 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1247 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1259 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1347 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1369 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1316 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1370 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1286 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1311 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:24,091 | test | INFO | pool-27-thread-9 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-728 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-727 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-713 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-813 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-755 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-789 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-710 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-787 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-886 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-860 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:26,441 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 88.46 2021-05-17 11:53:26,694 | test | INFO | pool-27-thread-16 | [table_view:display:72] Keys failed in default:scope2:fkPvY9-56-293000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1402 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1392 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:29,634 | test | INFO | pool-27-thread-27 | [table_view:display:72] Keys failed in default:scope2:KnehItU3q2Cui-56-292000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1402 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:31,548 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 90.84 2021-05-17 11:53:31,680 | test | INFO | pool-27-thread-3 | [table_view:display:72] Keys failed in default:_default:2_A9Wug-56-287000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1182 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1193 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1214 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1369 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1338 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1370 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1260 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1220 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1199 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1277 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1365 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1200 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:36,654 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 93.66 2021-05-17 11:53:37,191 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in default:_default:NMDV4ujHyTT-56-287000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-669 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-648 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-659 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-527 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-568 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-577 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-654 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-500 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-633 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-686 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-693 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:39,891 | test | INFO | pool-27-thread-13 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1433 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1415 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1439 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1384 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1430 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:41,164 | test | INFO | pool-27-thread-9 | [table_view:display:72] Keys failed in default:_default:Xb7ESVXH_HVybOv-56-287000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-908 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-955 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-950 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:41,740 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 95.94 2021-05-17 11:53:42,055 | test | INFO | pool-27-thread-25 | [table_view:display:72] Keys failed in default:dJn7gcoH7FzbN87CbkQ-55-103000:aw3IFm-e%ur-56-404000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:42,055 | test | INFO | pool-27-thread-25 | [table_view:display:72] Keys failed in default:dJn7gcoH7FzbN87CbkQ-55-103000:aw3IFm-e%ur-56-404000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-700 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-690 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:46,194 | test | INFO | pool-27-thread-30 | [table_view:display:72] Keys failed in default:Jv-56-613000:FpSTty6PF-56-771000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1905 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1895 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:46,194 | test | INFO | pool-27-thread-30 | [table_view:display:72] Keys failed in default:Jv-56-613000:FpSTty6PF-56-771000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1905 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1895 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:46,200 | test | INFO | pool-27-thread-4 | [table_view:display:72] Keys failed in default:Jv-56-613000:Aj-56-770000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1905 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1895 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:46,200 | test | INFO | pool-27-thread-4 | [table_view:display:72] Keys failed in default:Jv-56-613000:Aj-56-770000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1905 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1895 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:46,842 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 98.21 2021-05-17 11:53:47,426 | test | INFO | pool-27-thread-3 | [table_view:display:72] Keys failed in default:_default:2_A9Wug-56-287000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1439 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1407 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1432 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:51,973 | test | INFO | pool-27-thread-14 | [table_view:display:72] Keys failed in default:_default:zPdIU2SXD_SAHMz%-56-288000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-995 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1136 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1026 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1128 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1029 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1007 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1129 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1041 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1153 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1142 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1120 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1165 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1033 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1034 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1133 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1145 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:53:52,148 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 11:53:54,622 | test | INFO | pool-27-thread-29 | [table_view:display:72] Keys failed in default:_default:NMDV4ujHyTT-56-287000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-819 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-827 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-838 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-805 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-824 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-747 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-810 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-788 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-766 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-773 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-861 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-840 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-893 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:54:02,088 | test | INFO | pool-27-thread-3 | [table_view:display:72] Keys failed in default:_default:bTKZaX2-56-287000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-989 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-995 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1039 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 11:54:02,260 | test | INFO | pool-27-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 11:54:02,358 | test | INFO | pool-27-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 193.844000101 sec 2021-05-17 11:57:56,438 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 11:57:56,441 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 11:57:56,444 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 11:57:56,446 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 11:57:56,447 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 11:57:56,447 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 11:57:56,448 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 11:58:34,489 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 48 docs from 1000 to 1960 2021-05-17 11:58:34,490 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 2 docs from 980 to 1440 2021-05-17 11:58:34,493 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 8 docs from 980 to 1440 2021-05-17 11:59:43,240 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 11:59:43,240 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 17 docs from 500 to 980 2021-05-17 11:59:43,242 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 18 docs from 980 to 1440 2021-05-17 11:59:43,242 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 14 docs from 980 to 1440 2021-05-17 11:59:43,243 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 19 docs from 980 to 1440 2021-05-17 11:59:43,243 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 33 docs from 980 to 1440 2021-05-17 11:59:43,244 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 34 docs from 500 to 980 2021-05-17 11:59:43,246 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 41 docs from 980 to 1440 2021-05-17 11:59:43,246 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 33 docs from 500 to 980 2021-05-17 11:59:43,247 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 18 docs from 980 to 1440 2021-05-17 12:01:06,167 | test | ERROR | pool-27-thread-22 | [task:call:3809] Failed to load 4 docs from 980 to 1440 2021-05-17 12:04:19,710 | test | INFO | pool-27-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................****X | 196.827 | | 115.479 | .........................................................***X | 115.479 | | 107.977 | .....................................................X | 107.977 | | 79.492 | .......................................*****...X | 85.372 | | 85.370 | ..........................................****.........................................*.............***X | 192.514 | | 170.975 | .....................................................................................***X | 170.975 | | 112.166 | .......................................................X | 112.166 | | 52.585 | .........................****..............................................****...........*...........***X | 185.810 | | 107.032 | .....................................................X | 107.032 | | 82.757 | .........................................****...*****.........*****........****X | 122.666 | | 69.550 | ..................................*****.......................***X | 115.596 | | 112.195 | ........................................................X | 112.195 | | 94.271 | ..............................................*****........****X | 108.828 | | 92.853 | ..............................................X | 92.853 | | 77.395 | ......................................*****............................***X | 132.922 | | 132.922 | ..................................................................X | 132.922 | | 128.198 | ................................................................*........****X | 145.187 | | 109.906 | ......................................................****X | 109.906 | | 109.301 | ......................................................X | 109.301 | | 107.289 | .....................................................*****.*........****X | 125.754 | | 114.586 | .........................................................X | 114.586 | | 108.904 | ......................................................****X | 108.904 | | 81.143 | ........................................*X | 81.143 | | 81.143 | ........................................****........................****X | 128.410 | | 116.707 | ..........................................................X | 116.707 | | 110.803 | .......................................................*..****....*..*****.X | 129.167 | | 120.894 | ............................................................****X | 120.894 | | 102.693 | ...................................................X | 102.693 | | 92.793 | ..............................................****X | 92.793 | | 74.913 | .....................................*****.*.*****...........*****.....X | 111.027 | | 107.928 | .....................................................****X | 107.928 | | 56.807 | ...........................*****X | 56.807 | | 47.800 | .......................X | 47.800 | | 46.200 | ......................*****....****X | 53.308 | | 48.029 | .......................X | 48.029 | | 45.813 | ......................*****....*...........*****..............*****X | 103.815 | | 65.008 | ................................*****X | 65.008 | | 54.703 | ..........................******....*****..*****X | 65.505 | | 60.910 | .............................*****X | 60.910 | | 56.309 | ...........................*****X | 56.309 | | 54.205 | ..........................************.........****X | 72.799 | | 67.280 | .................................*****X | 67.889 | | 58.719 | ............................*****X | 58.719 | | 53.198 | ..........................******.....*****X | 64.104 | | 46.510 | ......................*****X | 46.510 | | 0.800 | *****.*.*****........................*****X | 52.101 | | 46.508 | ......................*****X | 46.508 | | 2.599 | ***********.........*..............................****X | 78.790 | | 43.506 | .....................X | 43.506 | | 6.801 | ..****X | 6.801 | | 0.000 | ******..******.................................*****X | 69.007 | | 65.098 | ................................*****X | 65.098 | | 47.803 | .......................****X | 47.803 | | 35.111 | ................X | 35.111 | | 0.300 | *****.....*..................*****..*....****X | 58.391 | | 21.506 | .........******..............****X | 48.800 | | 40.406 | ...................X | 40.406 | | 23.606 | ..........X | 23.606 | | 18.301 | ........****.........*..*****..****X | 43.904 | | 23.610 | ..........X | 23.610 | | 5.003 | .****X | 5.003 | | 3.803 | X | 3.803 | | 3.802 | ****X | 3.802 | | 1.400 | ****X | 1.400 | | 1.201 | X | 1.201 | | 0.000 | ******X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:04:20,736 | test | INFO | pool-27-thread-28 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................***X | 809.690 | | 385.693 | ..............................................X | 385.693 | | 249.854 | .............................***X | 249.854 | | 222.665 | ..........................X | 222.665 | | 213.484 | .........................***X | 213.484 | | 198.528 | .......................*****.............*....***X | 332.825 | | 187.172 | ......................****......*......***X | 286.521 | | 182.493 | .....................****.........*.........***........*..........***X | 476.716 | | 375.617 | .............................................X | 375.617 | | 239.971 | ............................***X | 239.971 | | 228.864 | ...........................*****X | 232.261 | | 193.976 | ......................X | 193.976 | | 183.554 | .....................***......*..***X | 245.934 | | 124.900 | ..............X | 124.900 | | 94.296 | ..........****....*****.........*........***X | 260.876 | | 231.267 | ...........................X | 231.267 | | 187.767 | ......................X | 187.767 | | 169.610 | ...................****....*....X | 229.285 | | 187.119 | ......................***X | 187.119 | | 132.906 | ...............*.****..*.....****X | 198.621 | | 191.311 | ......................X | 191.311 | | 168.818 | ...................*****..*..****X | 194.696 | | 166.906 | ...................*****X | 166.906 | | 165.122 | ...................X | 165.122 | | 149.982 | .................****X | 149.982 | | 134.239 | ...............X | 134.239 | | 127.476 | ..............***X | 127.476 | | 123.772 | ..............******...*.****X | 156.806 | | 136.158 | ...............*****..X | 150.022 | | 143.920 | ................*****.....****X | 183.936 | | 175.394 | ....................X | 175.394 | | 166.593 | ...................****X | 166.593 | | 144.059 | ................******.....****X | 182.914 | | 175.813 | ....................X | 175.813 | | 174.514 | ....................*****X | 174.905 | | 173.205 | ....................****X | 173.205 | | 149.840 | .................X | 149.840 | | 143.630 | ................*****..*...****X | 182.250 | | 148.324 | .................******.*****.....*..****X | 217.704 | | 192.330 | ......................******.....****X | 229.361 | | 159.802 | ..................******....*.***X | 197.048 | | 151.458 | .................X | 151.458 | | 122.210 | ..............******.*****.*.****X | 152.619 | | 138.788 | ................*****....*****X | 175.560 | | 166.956 | ...................******..****X | 179.187 | | 176.386 | ....................*....*****.....*...*****...........****X | 360.911 | | 320.279 | ......................................X | 320.279 | | 200.314 | .......................****X | 200.314 | | 173.033 | ....................X | 173.033 | | 139.395 | ................*****..*......*****.........*....*****......*..........****X | 440.778 | | 174.281 | ....................******..........................*****X | 388.299 | | 308.275 | .....................................****X | 308.275 | | 296.074 | ...................................X | 296.074 | | 232.326 | ...........................*****..*.....****X | 283.638 | | 271.815 | ................................X | 271.815 | | 203.976 | ........................****X | 203.976 | | 185.714 | .....................X | 185.714 | | 172.418 | ....................****X | 172.418 | | 159.099 | ..................*..*********X | 173.411 | | 168.706 | ...................*.*****...*.****X | 207.032 | | 169.339 | ...................*.X | 171.939 | | 159.900 | ..................****.....*..****X | 218.398 | | 188.270 | ......................X | 188.270 | | 186.369 | ......................****X | 186.369 | | 182.818 | .....................*****....*....******......****X | 298.618 | | 283.207 | .................................*****....*...*...***X | 358.907 | | 318.834 | ......................................X | 318.834 | | 212.119 | .........................*X | 212.119 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:04:21,135 | test | INFO | pool-27-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.......................*.......................****.X | 99.479 | | 97.985 | ..............................................****X | 98.190 | | 97.794 | .............................................***X | 97.794 | | 20.294 | ........*****................................................****X | 120.511 | | 102.000 | ...............................................X | 102.000 | | 45.794 | ....................**X | 45.794 | | 45.794 | ....................X | 45.794 | | 42.992 | ...................X | 42.992 | | 41.688 | ...................***X | 41.688 | | 30.380 | .............****........*.....****...****.X | 65.743 | | 63.090 | .............................*****..****X | 68.289 | | 65.793 | ..............................*.**X | 67.982 | | 64.641 | ..............................X | 64.641 | | 53.160 | ........................****......*.....****..*..****.....*****..*......*****.............****X | 139.452 | | 115.268 | ......................................................X | 115.268 | | 83.968 | .......................................*****............******................................................****X | 208.379 | | 183.384 | .......................................................................................X | 183.384 | | 103.086 | ................................................*****.*.......*****..*...****X | 130.696 | | 129.390 | .............................................................X | 129.390 | | 109.609 | ...................................................****X | 109.609 | | 108.506 | ...................................................*.......****X | 124.719 | | 110.022 | ...................................................******................****X | 143.253 | | 126.325 | ...........................................................X | 126.325 | | 115.126 | ......................................................****X | 115.126 | | 100.915 | ...............................................****X | 100.915 | | 99.304 | ..............................................******........****X | 115.829 | | 93.627 | ...........................................X | 93.627 | | 93.027 | ...........................................*****............................................*****X | 184.247 | | 137.518 | ................................................................X | 137.518 | | 71.902 | .................................*****.................*****X | 107.310 | | 86.813 | ........................................*****.X | 88.008 | | 81.128 | .....................................****X | 81.128 | | 45.003 | ....................*****...................................*............****X | 142.705 | | 59.206 | ...........................*****X | 59.206 | | 43.308 | ...................*****.......*****X | 57.999 | | 47.206 | .....................X | 47.206 | | 42.811 | ...................****..*..*****......*......****X | 75.110 | | 50.605 | .......................*****X | 50.605 | | 48.594 | ......................****X | 48.594 | | 45.791 | ....................X | 45.791 | | 39.897 | ..................*****..*.....****X | 55.918 | | 54.211 | .........................X | 54.211 | | 41.896 | ...................****X | 41.896 | | 31.896 | ..............X | 31.896 | | 3.301 | ****X | 3.301 | | 2.500 | ******............*...............****.*.****X | 63.905 | | 63.106 | .............................X | 63.106 | | 62.498 | ............................****X | 62.498 | | 3.200 | *****X | 3.200 | | 2.500 | X | 2.500 | | 1.801 | *****.*****X | 2.402 | | 1.400 | ******.*****...................................****X | 75.321 | | 67.325 | ...............................X | 67.325 | | 24.503 | ..........****X | 24.503 | | 10.608 | ....X | 10.608 | | 0.000 | **********X | 2.000 | | 0.800 | X | 0.800 | | 0.200 | *****.........................................****X | 85.899 | | 84.306 | .......................................X | 84.306 | | 75.704 | ...................................****X | 75.704 | | 60.206 | ...........................X | 60.206 | | 42.194 | ...................*****...............*****X | 73.219 | | 65.003 | ..............................******..*****....*.....*****...........*****X | 111.008 | | 79.414 | .....................................******................*****X | 112.606 | | 83.603 | .......................................****X | 83.603 | | 62.996 | .............................X | 62.996 | | 33.890 | ...............****X | 33.890 | | 15.104 | ......X | 15.104 | | 0.000 | *X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:04:21,188 | test | WARNING | Thread-88934 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:tXD-25-280000:782d661_URgFQ5n-28-290000 update failed keys from task: [u'test_collections-1847'] 2021-05-17 12:04:21,191 | test | WARNING | Thread-88943 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:1cbvs51B-24-617000:fiVH-24-769000 update failed keys from task: [u'test_collections-1847'] 2021-05-17 12:04:21,191 | test | WARNING | Thread-88941 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:a0k-24-961000:pNIeqrGWM0Ondg-25-130000 update failed keys from task: [u'test_collections-1109', u'test_collections-1183', u'test_collections-1341', u'test_collections-1578', u'test_collections-1172', u'test_collections-1176', u'test_collections-1056', u'test_collections-1057', u'test_collections-1444', u'test_collections-1566', u'test_collections-1688', u'test_collections-1284', u'test_collections-1440', u'test_collections-1200', u'test_collections-1442', u'test_collections-1201', u'test_collections-1281', u'test_collections-1718', u'test_collections-1554', u'test_collections-1314', u'test_collections-1559', u'test_collections-1030', u'test_collections-1311', u'test_collections-1390', u'test_collections-1391', u'test_collections-1270', u'test_collections-1269', u'test_collections-1423', u'test_collections-1149', u'test_collections-1020', u'test_collections-1378', u'test_collections-1533', u'test_collections-1099', u'test_collections-1133', u'test_collections-1013', u'test_collections-1488', u'test_collections-1523', u'test_collections-1407', u'test_collections-1087', u'test_collections-1485', u'test_collections-1124', u'test_collections-1518', u'test_collections-1235', u'test_collections-1117', u'test_collections-1515', u'test_collections-1075', u'test_collections-1354', u'test_collections-1597'] 2021-05-17 12:04:21,193 | test | WARNING | Thread-88939 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:PqiRsn8V1jAjlAGRgLHI-53-0:SHavScxByA-53-142000 create failed keys from task: [u'test_collections-1718', u'test_collections-1688'] 2021-05-17 12:04:21,193 | test | WARNING | Thread-88945 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:eI8-53-338000:Ko-LKv9-53-451000 create failed keys from task: [u'test_collections-1718', u'test_collections-1688'] 2021-05-17 12:04:21,217 | test | WARNING | Thread-89046 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:EgDE-56-288000 update failed keys from task: [u'test_collections-1257', u'test_collections-1325', u'test_collections-1414', u'test_collections-1217', u'test_collections-1418', u'test_collections-1429', u'test_collections-1186', u'test_collections-1363'] 2021-05-17 12:04:21,217 | test | WARNING | Thread-89051 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:awlCa-56-289000 update failed keys from task: [u'test_collections-1411', u'test_collections-1387'] 2021-05-17 12:04:21,220 | test | WARNING | Thread-89055 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Xb7ESVXH_HVybOv-56-287000 update failed keys from task: [u'test_collections-1071', u'test_collections-1391', u'test_collections-1224', u'test_collections-1225', u'test_collections-1423', u'test_collections-1347', u'test_collections-1308', u'test_collections-1183', u'test_collections-1020', u'test_collections-1384', u'test_collections-1022', u'test_collections-1220', u'test_collections-1016', u'test_collections-1335', u'test_collections-1259', u'test_collections-1415', u'test_collections-1370', u'test_collections-1010', u'test_collections-1298', u'test_collections-988', u'test_collections-1202', u'test_collections-1247', u'test_collections-1369', u'test_collections-1286', u'test_collections-1281', u'test_collections-993', u'test_collections-1433', u'test_collections-994', u'test_collections-1316', u'test_collections-1439', u'test_collections-1275', u'test_collections-1430', u'test_collections-1311'] 2021-05-17 12:04:21,217 | test | WARNING | Thread-89049 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:NMDV4ujHyTT-56-287000 delete failed keys from task: [u'test_collections-805', u'test_collections-606', u'test_collections-648', u'test_collections-527', u'test_collections-568', u'test_collections-766', u'test_collections-569', u'test_collections-885', u'test_collections-686', u'test_collections-840', u'test_collections-819', u'test_collections-659', u'test_collections-656', u'test_collections-810', u'test_collections-577', u'test_collections-654', u'test_collections-773', u'test_collections-893', u'test_collections-693', u'test_collections-827', u'test_collections-627', u'test_collections-669', u'test_collections-824', u'test_collections-747', u'test_collections-788', u'test_collections-500', u'test_collections-861', u'test_collections-662', u'test_collections-838', u'test_collections-518', u'test_collections-632', u'test_collections-633', u'test_collections-795'] 2021-05-17 12:04:21,217 | test | WARNING | Thread-89047 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:zPdIU2SXD_SAHMz%-56-288000 update failed keys from task: [u'test_collections-1026', u'test_collections-980', u'test_collections-1128', u'test_collections-1029', u'test_collections-1007', u'test_collections-1129', u'test_collections-1041', u'test_collections-1142', u'test_collections-1120', u'test_collections-1165', u'test_collections-1145', u'test_collections-995', u'test_collections-1136', u'test_collections-1019', u'test_collections-1153', u'test_collections-1033', u'test_collections-1034', u'test_collections-1133'] 2021-05-17 12:04:21,224 | test | WARNING | Thread-89068 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:bTKZaX2-56-287000 update failed keys from task: [u'test_collections-989', u'test_collections-995', u'test_collections-1039', u'test_collections-1008'] 2021-05-17 12:04:21,224 | test | WARNING | Thread-89069 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:2_A9Wug-56-287000 update failed keys from task: [u'test_collections-1193', u'test_collections-1103', u'test_collections-1423', u'test_collections-1149', u'test_collections-1106', u'test_collections-1307', u'test_collections-1428', u'test_collections-1062', u'test_collections-1260', u'test_collections-1220', u'test_collections-1182', u'test_collections-1059', u'test_collections-1214', u'test_collections-1338', u'test_collections-1370', u'test_collections-1096', u'test_collections-1055', u'test_collections-1297', u'test_collections-1056', u'test_collections-1093', u'test_collections-1050', u'test_collections-1005', u'test_collections-1369', u'test_collections-1407', u'test_collections-1087', u'test_collections-1365', u'test_collections-1200', u'test_collections-1201', u'test_collections-1080', u'test_collections-1081', u'test_collections-991', u'test_collections-1117', u'test_collections-1439', u'test_collections-1152', u'test_collections-1075', u'test_collections-1110', u'test_collections-1275', u'test_collections-1111', u'test_collections-1199', u'test_collections-1277', u'test_collections-1432'] 2021-05-17 12:04:21,230 | test | WARNING | Thread-89081 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Jv-56-613000:Aj-56-770000 create failed keys from task: [u'test_collections-1905', u'test_collections-1895'] 2021-05-17 12:04:21,240 | test | WARNING | Thread-89112 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:fkPvY9-56-293000 update failed keys from task: [u'test_collections-1292', u'test_collections-1302', u'test_collections-1402', u'test_collections-1227', u'test_collections-1249', u'test_collections-1360', u'test_collections-1241', u'test_collections-1266', u'test_collections-1223', u'test_collections-1366', u'test_collections-1180', u'test_collections-1279', u'test_collections-1237', u'test_collections-1435', u'test_collections-1239', u'test_collections-1392', u'test_collections-1230', u'test_collections-1252', u'test_collections-1353'] 2021-05-17 12:04:21,240 | test | WARNING | Thread-89118 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:WmJYX2vN-56-293000 delete failed keys from task: [u'test_collections-907', u'test_collections-948', u'test_collections-969', u'test_collections-863', u'test_collections-740', u'test_collections-782', u'test_collections-780', u'test_collections-818', u'test_collections-939', u'test_collections-715', u'test_collections-836', u'test_collections-914', u'test_collections-812', u'test_collections-977', u'test_collections-833', u'test_collections-954', u'test_collections-750'] 2021-05-17 12:04:21,243 | test | WARNING | Thread-89123 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:-gEf1vIqdx4dzA-56-273000 delete failed keys from task: [u'test_collections-4'] 2021-05-17 12:04:21,266 | test | WARNING | Thread-89193 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ToVxURDN%t3Wk6t-53-769000:ibYsI3-53-898000 create failed keys from task: [u'test_collections-456'] 2021-05-17 12:04:21,312 | test | WARNING | Thread-89113 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:KnehItU3q2Cui-56-292000 update failed keys from task: [u'test_collections-1292', u'test_collections-1302', u'test_collections-1226', u'test_collections-1402', u'test_collections-1249', u'test_collections-1360', u'test_collections-1241', u'test_collections-1266', u'test_collections-1366', u'test_collections-1180', u'test_collections-1237', u'test_collections-1435', u'test_collections-1252', u'test_collections-1353'] 2021-05-17 12:04:21,315 | test | WARNING | Thread-89119 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:A6MOecKi__pz-56-293000 update failed keys from task: [u'test_collections-1292', u'test_collections-1302', u'test_collections-1204', u'test_collections-1402', u'test_collections-1249', u'test_collections-1360', u'test_collections-1186', u'test_collections-1244', u'test_collections-1266', u'test_collections-1223', u'test_collections-1180', u'test_collections-1279', u'test_collections-1435', u'test_collections-1218', u'test_collections-1392', u'test_collections-1230', u'test_collections-1252', u'test_collections-1353'] 2021-05-17 12:04:21,328 | test | WARNING | Thread-89085 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:dJn7gcoH7FzbN87CbkQ-55-103000:aw3IFm-e%ur-56-404000 create failed keys from task: [u'test_collections-700', u'test_collections-690'] 2021-05-17 12:04:21,339 | test | WARNING | Thread-89080 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Jv-56-613000:FpSTty6PF-56-771000 create failed keys from task: [u'test_collections-1905', u'test_collections-1576', u'test_collections-1895', u'test_collections-1536'] 2021-05-17 12:04:22,737 | test | WARNING | Thread-89055 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Xb7ESVXH_HVybOv-56-287000 delete failed keys from task: [u'test_collections-728', u'test_collections-529', u'test_collections-848', u'test_collections-727', u'test_collections-604', u'test_collections-689', u'test_collections-567', u'test_collections-886', u'test_collections-539', u'test_collections-979', u'test_collections-614', u'test_collections-813', u'test_collections-612', u'test_collections-610', u'test_collections-655', u'test_collections-774', u'test_collections-908', u'test_collections-509', u'test_collections-625', u'test_collections-901', u'test_collections-789', u'test_collections-786', u'test_collections-787', u'test_collections-661', u'test_collections-783', u'test_collections-860', u'test_collections-713', u'test_collections-955', u'test_collections-678', u'test_collections-755', u'test_collections-558', u'test_collections-710', u'test_collections-598', u'test_collections-950'] 2021-05-17 12:04:33,648 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 12:05:35,127 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 12:05:35,128 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 12:05:59,148 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 12:05:59,148 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 12:06:20,246 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 12:06:20,246 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 12:07:10,571 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1340000629 seconds to complete 2021-05-17 12:07:10,586 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3337.178s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 25 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_25 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_26 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_26', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 26, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 12:07:20,036 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #26 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 12:07:20,710 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #26 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 12:07:21,226 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xedbf97 type=client open_count=1 channel=[id: 0xb1aca31e, 0.0.0.0/0.0.0.0:46309] timeout=300.0>) 2021-05-17 12:07:42,003 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xedbf98 type=client open_count=1 channel=[id: 0x47762ba7, 0.0.0.0/0.0.0.0:43687] timeout=300.0>) 2021-05-17 12:07:44,035 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xedbf99 type=client open_count=1 channel=[id: 0x89223bdf, 0.0.0.0/0.0.0.0:35057] timeout=300.0>) 2021-05-17 12:07:46,052 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xedbf9a type=client open_count=1 channel=[id: 0x613cae09, 0.0.0.0/0.0.0.0:54950] timeout=300.0>) 2021-05-17 12:08:10,003 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xedbf9b type=client open_count=1 channel=[id: 0x19dacb87, 0.0.0.0/0.0.0.0:55876] timeout=300.0>) 2021-05-17 12:08:12,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xedbf9c type=client open_count=1 channel=[id: 0x845970cd, 0.0.0.0/0.0.0.0:50852] timeout=300.0>) 2021-05-17 12:08:14,043 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xedbf9d type=client open_count=1 channel=[id: 0x3a47db47, 0.0.0.0/0.0.0.0:33020] timeout=300.0>) 2021-05-17 12:08:37,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xedbf9e type=client open_count=1 channel=[id: 0xf5c135cd, 0.0.0.0/0.0.0.0:54045] timeout=300.0>) 2021-05-17 12:08:39,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xedbf9f type=client open_count=1 channel=[id: 0x823a246f, 0.0.0.0/0.0.0.0:57117] timeout=300.0>) 2021-05-17 12:08:42,006 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xedbfa0 type=client open_count=1 channel=[id: 0x80a233ef, 0.0.0.0/0.0.0.0:49953] timeout=300.0>) 2021-05-17 12:09:06,062 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xedbfa1 type=client open_count=1 channel=[id: 0xf2dc1fb9, 0.0.0.0/0.0.0.0:35089] timeout=300.0>) 2021-05-17 12:09:08,079 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xedbfa2 type=client open_count=1 channel=[id: 0xe285386c, 0.0.0.0/0.0.0.0:58625] timeout=300.0>) 2021-05-17 12:09:10,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xedbfa3 type=client open_count=1 channel=[id: 0x9488395c, 0.0.0.0/0.0.0.0:44780] timeout=300.0>) 2021-05-17 12:09:34,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xedbfa4 type=client open_count=1 channel=[id: 0xfc6a10cd, 0.0.0.0/0.0.0.0:60975] timeout=300.0>) 2021-05-17 12:09:36,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xedbfa5 type=client open_count=1 channel=[id: 0x178b4aad, 0.0.0.0/0.0.0.0:37812] timeout=300.0>) 2021-05-17 12:09:38,681 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 12:09:55,927 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 12:10:09,632 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 12:10:09,657 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #26 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 12:10:09,694 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #26 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 12:10:23,144 | test | INFO | pool-28-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.79333333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 12:10:23,167 | test | INFO | pool-28-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 12:10:33,243 | test | INFO | pool-28-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 12:10:33,269 | test | INFO | pool-28-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1270000935 sec 2021-05-17 12:10:33,556 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 12:11:28,753 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 12:11:29,132 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 622615912 | 100301431 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 665593536 | 114189241 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 611577608 | 88814242 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 12:11:29,134 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 12:12:14,775 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 12:12:14,785 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 12:29:20,687 | test | INFO | pool-28-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******....................*................................................................................****X | 904.443 | | 661.520 | ........................................................................****X | 661.520 | | 611.088 | ..................................................................X | 611.088 | | 596.700 | ................................................................****X | 596.700 | | 538.550 | ..........................................................******.******..........*****X | 639.774 | | 631.924 | ....................................................................****X | 631.924 | | 594.313 | ................................................................****X | 594.313 | | 594.012 | ................................................................*...***X | 619.432 | | 581.750 | ...............................................................X | 581.750 | | 525.441 | .........................................................****X | 525.441 | | 480.697 | ....................................................******............*****.X | 605.587 | | 573.397 | ..............................................................****X | 573.397 | | 500.573 | ......................................................*****X | 500.573 | | 486.538 | ....................................................****X | 486.538 | | 463.346 | ..................................................X | 463.346 | | 457.556 | .................................................*****........*****X | 531.171 | | 525.491 | .........................................................****X | 525.491 | | 525.091 | .........................................................X | 525.091 | | 517.334 | ........................................................*****......*.****X | 582.974 | | 523.877 | ........................................................*****X | 523.877 | | 445.876 | ................................................*****.*...*.****.X | 501.816 | | 483.074 | ....................................................*****..****X | 498.002 | | 490.302 | .....................................................X | 490.302 | | 467.454 | ..................................................*...****...*.****X | 529.012 | | 485.692 | ....................................................*****.*..*****..X | 525.778 | | 414.550 | ............................................****...*...*****X | 469.705 | | 450.854 | ................................................*****.........*****X | 529.853 | | 418.214 | .............................................******........*****...*****X | 517.403 | | 414.511 | ............................................*****.X | 420.478 | | 410.515 | ............................................**********X | 412.662 | | 410.065 | ............................................*****..*.......*****...*****X | 523.821 | | 456.444 | .................................................*****.......****X | 519.026 | | 404.561 | ...........................................******..******..............****X | 549.577 | | 535.880 | ..........................................................X | 535.880 | | 485.517 | ....................................................****X | 485.517 | | 376.448 | ........................................*****...*..........***X | 488.730 | | 483.297 | ....................................................X | 483.297 | | 471.082 | ...................................................*****.*****X | 486.056 | | 468.518 | ..................................................*****..*****..*.****X | 509.373 | | 484.922 | ....................................................*****..X | 504.124 | | 487.113 | ....................................................*****.*......****X | 545.494 | | 397.609 | ..........................................******...*****.....*****X | 464.996 | | 463.551 | ..................................................*****X | 463.551 | | 430.813 | ..............................................*****.....*.....****X | 516.264 | | 461.014 | .................................................*****X | 461.014 | | 365.623 | .......................................*****...........*****X | 469.521 | | 423.755 | .............................................*****.*.****X | 442.611 | | 440.656 | ...............................................X | 440.656 | | 437.963 | ...............................................*****..****X | 457.110 | | 352.014 | .....................................X | 352.014 | | 332.054 | ...................................*****..........*****..*..........****X | 529.557 | | 454.157 | .................................................****X | 454.157 | | 453.970 | .................................................*****X | 455.733 | | 350.998 | .....................................******..............*****.*****X | 484.282 | | 411.498 | ............................................*****.******......****X | 478.903 | | 441.889 | ...............................................X | 441.889 | | 410.784 | ............................................*****.......****X | 476.922 | | 474.552 | ...................................................*..****X | 491.091 | | 468.658 | ..................................................X | 468.658 | | 446.797 | ................................................*****X | 451.052 | | 447.152 | ................................................****...*........****X | 549.903 | | 490.330 | .....................................................****X | 490.330 | | 489.918 | .....................................................X | 489.918 | | 488.477 | .....................................................****X | 488.477 | | 433.790 | ..............................................******......X | 487.474 | | 483.867 | ....................................................***X | 483.867 | | 458.129 | .................................................X | 458.129 | | 397.414 | ..........................................*****.*...*****........****X | 503.233 | | 465.217 | ..................................................*****X | 465.217 | | 454.278 | .................................................*****X | 454.278 | | 401.733 | ...........................................*****..........******.******..............******.....*X | 672.706 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:29:21,556 | test | INFO | pool-28-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ********************************************************.......*.......*****...............*****X | 5.999 | | 0.000 | ***********...*....*****...*............****X | 4.500 | | 1.200 | ....X | 1.200 | | 0.000 | **********..........*.......................................****X | 10.100 | | 7.887 | ......................................*****X | 7.887 | | 1.498 | ......*****X | 1.498 | | 0.000 | *****................................................*****X | 9.887 | | 3.596 | ................****X | 3.596 | | 2.796 | ............X | 2.796 | | 0.000 | *****.......*****X | 1.499 | | 0.399 | X | 0.399 | | 0.000 | **********.....................................*****X | 7.491 | | 0.000 | ******..............****X | 2.997 | | 1.499 | ......******...............****X | 4.497 | | 4.297 | ....................X | 4.297 | | 1.499 | ......*****............*..........****X | 5.995 | | 5.095 | ........................X | 5.095 | | 1.499 | ......****X | 1.499 | | 0.000 | *****....*..................****X | 4.496 | | 0.000 | *****.*......*****..................................................................................*****X | 17.993 | | 0.000 | ***********...................................................*****X | 10.495 | | 0.000 | *****...........*...**********X | 2.999 | | 0.000 | *****..*............*****.......................****X | 7.498 | | 1.501 | ......X | 1.501 | | 0.000 | *****...*..........****X | 2.699 | | 0.300 | ****X | 0.300 | | 0.000 | ******.......*****.......******.......................****X | 7.498 | | 2.700 | ............X | 2.700 | | 1.500 | ......****X | 1.500 | | 0.000 | *****...............*.....*****....******.....****X | 6.000 | | 5.199 | ........................X | 5.199 | | 4.499 | .....................****X | 4.499 | | 4.199 | ...................X | 4.199 | | 2.999 | .............****X | 2.999 | | 1.500 | ......*****X | 1.500 | | 0.000 | *****...................................................................................................*****X | 20.198 | | 5.299 | .........................****X | 5.299 | | 4.899 | .......................X | 4.899 | | 0.000 | *****.*.........*****.*****X | 2.299 | | 0.000 | *****............................................*****X | 8.998 | | 4.500 | .....................****X | 4.500 | | 3.600 | ................X | 3.600 | | 0.000 | *****..............*****............******..........................................................................****X | 20.200 | | 3.027 | .............*****X | 3.027 | | 0.000 | **********............*........................****X | 7.290 | | 3.992 | ..................X | 3.992 | | 3.399 | ...............*****..............*..................................****X | 13.094 | | 2.999 | .............*****X | 2.999 | | 0.000 | *****...........*......****X | 3.603 | | 2.401 | ..........****X | 2.401 | | 0.402 | X | 0.402 | | 0.000 | ****************....................................................****X | 10.506 | | 0.000 | ***********........*..................................****X | 8.609 | | 6.402 | ..............................****X | 6.402 | | 2.601 | ...........X | 2.601 | | 0.000 | ************************************X | 0.000 | +-------+---------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 12:29:22,555 | test | INFO | pool-28-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******...................................................................................*****X | 15.000 | | 0.000 | ***************************************......*..****X | 1.500 | | 0.000 | ******........**********X | 1.500 | | 1.200 | .....X | 1.200 | | 0.000 | *********************************........*.................................****X | 7.491 | | 2.996 | ...............*****X | 2.996 | | 0.000 | *****.............*...........****X | 4.494 | | 0.000 | ******........*****............*....****X | 4.496 | | 0.000 | ***********.*.................*......***X | 4.495 | | 2.095 | ..........X | 2.095 | | 0.000 | *****........***********................*****X | 4.497 | | 4.497 | .......................*****...*.......................................****X | 11.990 | | 6.391 | ..................................X | 6.391 | | 1.499 | .......****X | 1.499 | | 0.000 | *****.*.......*********X | 1.500 | | 0.000 | ******..................................................................****X | 11.995 | | 9.595 | ....................................................X | 9.595 | | 0.000 | ***********.........................................................................................****X | 16.192 | | 1.799 | ........******..............................****X | 7.196 | | 6.796 | ....................................X | 6.796 | | 6.298 | .................................****X | 6.298 | | 1.399 | ......******..........******.......****X | 4.499 | | 3.598 | ..................X | 3.598 | | 0.000 | *****................****X | 3.000 | | 1.499 | .......******.........................*****..............*****X | 8.597 | | 7.598 | .........................................****X | 7.598 | | 5.999 | ................................X | 5.999 | | 4.799 | .........................****X | 4.799 | | 3.599 | ..................X | 3.599 | | 0.000 | **********....................................................................................................*****X | 17.996 | | 2.998 | ...............******........*****.........................................*..........****X | 13.598 | | 4.499 | .......................*****X | 4.499 | | 0.000 | ***********................*****X | 3.000 | | 2.999 | ...............****X | 2.999 | | 1.500 | .......******........****X | 2.899 | | 1.500 | .......X | 1.500 | | 0.100 | ****X | 0.100 | | 0.000 | ******..........................................................*****X | 10.498 | | 3.000 | ...............****X | 3.000 | | 2.400 | ............X | 2.400 | | 0.000 | ****........*........****X | 3.000 | | 2.999 | ...............******..................................................****X | 12.000 | | 7.566 | .........................................*****X | 7.566 | | 1.500 | .......*****X | 1.500 | | 0.000 | ******.................................................................................****X | 14.579 | | 5.598 | ..............................*****X | 5.598 | | 4.898 | ..........................*****.....*.......****X | 7.197 | | 0.300 | ******........................****X | 4.502 | | 2.402 | ............X | 2.402 | | 0.000 | **********************..................................................****X | 9.005 | | 0.598 | ..X | 0.598 | | 0.000 | **********...*.....****X | 1.501 | | 0.000 | *******************************************X | 0.000 | +-------+----------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 12:29:22,811 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 78.6979782996 | 6437847040 | 4303495168 | 2432225280 / 7511588864 | 101534 / 205077 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 72.2047493052 | 6437847040 | 4287279104 | 2423431168 / 7511588864 | 98796 / 200998 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 80.6013433557 | 6437847040 | 4290555904 | 2398982144 / 7511588864 | 100914 / 204028 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 65.9893992933 | 6437847040 | 3839381504 | 3052236800 / 9530753024 | 101380 / 196919 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 71.8761979366 | 6437847040 | 4308013056 | 2409476096 / 7511588864 | 101131 / 200426 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 12:29:35,779 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 12:31:02,487 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162634520 | 268603587 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 161418032 | 284108770 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 443285176 | 436849414 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 12:31:02,489 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #26 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 12:31:12,369 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 12:31:12,371 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 12:31:12,371 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 12:31:22,778 | test | INFO | pool-28-thread-19 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.3180000782 seconds to complete 2021-05-17 12:31:22,782 | test | INFO | pool-28-thread-19 | [common_lib:sleep:22] Sleep 8.3180000782 seconds. Reason: Wait after rebalance complete 2021-05-17 12:33:31,180 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0759999752045 seconds 2021-05-17 12:33:31,180 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 12:33:31,181 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 12:33:39,321 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 12:45:58,029 | test | INFO | pool-28-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*****X | 1164.075 | | 465.068 | ......................................*****X | 465.068 | | 414.401 | ..................................******...............*..*X | 616.129 | | 616.129 | ...................................................*X | 616.129 | | 572.160 | ................................................X | 572.160 | | 543.036 | .............................................*****........*.....*......***X | 760.618 | | 629.561 | .....................................................X | 629.561 | | 558.143 | ..............................................X | 558.143 | | 454.905 | ......................................***X | 454.905 | | 362.513 | ..............................X | 362.513 | | 331.198 | ...........................X | 331.198 | | 314.877 | ..........................****........**...***X | 445.617 | | 330.435 | ...........................X | 330.435 | | 297.287 | ........................****X | 297.287 | | 277.022 | ......................X | 277.022 | | 267.517 | .....................****..**..***X | 302.976 | | 290.794 | .......................X | 290.794 | | 279.496 | .......................******.**X | 293.267 | | 293.267 | ........................**...........*....****X | 465.905 | | 197.473 | ...............*****..*......*****..*.*****......*...*X | 421.445 | | 421.445 | ...................................**X | 421.445 | | 285.889 | .......................X | 285.889 | | 229.662 | ..................*****..*..****X | 273.514 | | 245.596 | ....................X | 245.596 | | 239.988 | ...................*****...*.****X | 288.739 | | 202.487 | ................******...........*****X | 332.554 | | 287.240 | .......................******.........*****X | 391.380 | | 314.231 | .........................X | 314.231 | | 293.043 | ........................********X | 294.593 | | 294.593 | ........................*X | 294.593 | | 259.793 | .....................*****X | 259.793 | | 216.485 | .................*****X | 216.485 | | 208.611 | ................******...........*X | 330.283 | | 330.283 | ...........................*X | 330.283 | | 330.283 | ...........................**.....******.******..........**X | 515.291 | | 515.291 | ...........................................**X | 515.291 | | 428.774 | ...................................*****X | 428.774 | | 390.467 | ................................X | 390.467 | | 390.467 | ................................*X | 390.467 | | 390.467 | ................................***.*X | 404.759 | | 404.759 | .................................**X | 404.759 | | 404.759 | .................................X | 404.759 | | 322.986 | ..........................*****X | 322.986 | | 228.612 | ..................X | 228.612 | | 203.605 | ................*****....******..............*****X | 410.986 | | 366.136 | ..............................**X | 366.136 | | 366.136 | ..............................***.....****X | 423.322 | | 371.197 | ..............................X | 371.197 | | 339.268 | ............................*****X | 339.268 | | 273.348 | ......................******...............******............*..****X | 611.797 | | 584.017 | .................................................*****X | 584.017 | | 500.024 | .........................................***X | 500.024 | | 500.024 | .........................................*X | 500.024 | | 488.323 | ........................................*****X | 488.323 | | 448.246 | .....................................*****X | 448.246 | | 421.558 | ...................................X | 421.558 | | 416.647 | ..................................*****......*****X | 486.724 | | 421.260 | ...................................*****X | 421.260 | | 404.839 | .................................******........****X | 499.239 | | 465.799 | .......................................X | 465.799 | | 447.522 | .....................................*****...******..*****X | 506.051 | | 444.333 | .....................................*****X | 444.333 | | 404.111 | .................................*.*****..........*****X | 531.202 | | 498.074 | .........................................X | 498.074 | | 454.857 | ......................................*****....******..*****X | 525.502 | | 491.887 | .........................................******...****X | 527.573 | | 526.199 | ............................................X | 526.199 | | 495.152 | .........................................*****.*....****X | 553.670 | | 546.628 | .............................................X | 546.628 | | 539.278 | .............................................*****.......*****X | 621.330 | | 579.517 | ................................................X | 579.517 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 12:45:58,627 | test | INFO | pool-28-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..*****.......*...*....****....................*......****X | 52.699 | | 50.509 | .......................................******..........******..........*....*****....................*****X | 105.119 | | 96.693 | ............................................................................******.............*****......*....****X | 123.990 | | 122.706 | .................................................................................................X | 122.706 | | 119.204 | ...............................................................................................****X | 119.204 | | 106.688 | .....................................................................................******........*****X | 117.097 | | 113.781 | ..........................................................................................*****X | 113.781 | | 108.297 | ......................................................................................*****X | 108.297 | | 100.187 | ...............................................................................****X | 100.187 | | 97.182 | .............................................................................*..*****.............*.....****X | 121.885 | | 88.288 | ......................................................................X | 88.288 | | 83.386 | ..................................................................*****....*.................****X | 110.181 | | 104.103 | ..................................................................................*****X | 104.103 | | 82.292 | .................................................................*****..*...................****X | 108.378 | | 105.129 | ...................................................................................X | 105.129 | | 87.535 | .....................................................................****X | 87.535 | | 45.981 | ....................................******..******.....*....*****X | 60.298 | | 50.207 | .......................................******.*****X | 51.998 | | 44.698 | ...................................******.........*****X | 56.562 | | 5.803 | ...*****X | 5.803 | | 1.300 | ******.*****X | 2.500 | | 1.600 | X | 1.600 | | 1.200 | *****..*****X | 3.301 | | 1.500 | ***********X | 2.001 | | 1.200 | ******.....................*****...................*.....****X | 56.500 | | 50.003 | .......................................******................*****X | 69.714 | | 67.007 | .....................................................******........*****X | 77.603 | | 64.593 | ...................................................******....................******X | 89.418 | | 12.800 | .........*****X | 12.800 | | 2.800 | .*****X | 2.800 | | 1.700 | ****X | 1.700 | | 1.300 | ******.*.****X | 4.200 | | 2.099 | X | 2.099 | | 1.200 | ******.*****X | 2.101 | | 0.800 | ******.******.******.****X | 4.301 | | 2.700 | .X | 2.700 | | 1.700 | *****.*****X | 3.100 | | 1.700 | ******X | 2.000 | | 1.900 | ****X | 1.900 | | 1.900 | ************..*****X | 4.801 | | 2.100 | *****....*****X | 6.696 | | 3.302 | .*****X | 3.302 | | 1.800 | ******.*****X | 3.599 | | 3.501 | .******.*****X | 4.199 | | 2.600 | .*****X | 2.600 | | 1.200 | X | 1.200 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:45:59,999 | test | INFO | pool-28-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..*****.........*....*****.............*.....*****...........*....*****...*****X | 62.005 | | 59.893 | ................................................******......******.................*****X | 87.795 | | 85.503 | .....................................................................******........*.....****X | 100.696 | | 82.308 | ..................................................................X | 82.308 | | 76.205 | .............................................................****............*.......*......***X | 105.992 | | 92.405 | ...........................................................................X | 92.405 | | 76.899 | ..............................................................*****............*...****X | 95.490 | | 81.405 | ..................................................................******.....................*.****X | 107.887 | | 93.191 | ...........................................................................*....*****.....*.****X | 104.991 | | 97.291 | ...............................................................................X | 97.291 | | 96.587 | ..............................................................................*****.....................*****X | 121.182 | | 101.606 | ..................................................................................*****...X | 105.299 | | 103.994 | ....................................................................................****X | 103.994 | | 99.077 | ................................................................................**..****........******.....*****X | 116.851 | | 111.800 | ...........................................................................................X | 111.800 | | 106.453 | ......................................................................................****X | 106.453 | | 97.995 | ...............................................................................*****X | 97.995 | | 85.814 | .....................................................................******..............*****X | 102.497 | | 97.297 | ...............................................................................*****X | 97.297 | | 95.395 | .............................................................................X | 95.395 | | 82.457 | ...................................................................*****......******........X | 99.493 | | 97.896 | ...............................................................................****X | 97.896 | | 92.903 | ...........................................................................*..*****..................****X | 116.608 | | 116.608 | ...............................................................................................X | 116.608 | | 84.315 | ....................................................................*****X | 84.315 | | 68.794 | .......................................................******.......******....*****X | 81.993 | | 62.718 | ..................................................******.*****X | 63.102 | | 57.003 | ..............................................*****X | 57.003 | | 6.502 | ....*****X | 6.502 | | 2.400 | X | 2.400 | | 2.400 | ***X | 2.400 | | 2.400 | *.****X | 2.800 | | 2.000 | X | 2.000 | | 1.500 | *****X | 1.500 | | 1.300 | ******..*****X | 3.800 | | 3.100 | .***********X | 3.500 | | 3.100 | .*****X | 3.100 | | 2.600 | .***********X | 2.999 | | 2.701 | .*****X | 2.701 | | 2.700 | .*****X | 2.700 | | 0.500 | *******..*****..*****X | 5.501 | | 3.200 | .*******.****X | 4.100 | | 4.001 | ..*****X | 4.201 | | 2.900 | .X | 2.900 | | 2.100 | *****.*****X | 3.400 | | 2.501 | .******...*****X | 6.701 | | 2.900 | .*****..*****X | 5.097 | | 4.402 | ..X | 4.402 | | 4.202 | ..***X | 4.202 | | 3.000 | .X | 3.000 | | 2.500 | .******.***********..*****X | 6.799 | | 4.301 | ..*******X | 4.600 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 12:46:00,280 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 12:46:00,335 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 12:46:10,522 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 12:46:10,684 | test | INFO | pool-28-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.236650889941 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 48.4350221184 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 57.3197560081 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 55.5714809506 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 56.1461406404 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 12:46:10,714 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 12:46:16,844 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 12:46:16,976 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 12:46:21,940 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 2.44 2021-05-17 12:46:27,043 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 5.04 2021-05-17 12:46:32,115 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 7.86 2021-05-17 12:46:37,193 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 11.55 2021-05-17 12:46:42,292 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 15.83 2021-05-17 12:46:47,414 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 19.95 2021-05-17 12:46:52,536 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 23.74 2021-05-17 12:46:57,665 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 28.19 2021-05-17 12:47:02,769 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 32.36 2021-05-17 12:47:07,875 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 34.71 2021-05-17 12:47:10,226 | test | INFO | pool-28-thread-22 | [table_view:display:72] Keys failed in bucket1:nojN3VMu3H-15-414000:JOiuiPqs5-15-525000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:47:10,227 | test | INFO | pool-28-thread-22 | [table_view:display:72] Keys failed in bucket1:nojN3VMu3H-15-414000:JOiuiPqs5-15-525000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-174 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:47:10,232 | test | INFO | pool-28-thread-18 | [table_view:display:72] Keys failed in bucket1:HUgY-14-721000:jKWyjq-6bPZw-14-870000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:47:10,233 | test | INFO | pool-28-thread-18 | [table_view:display:72] Keys failed in bucket1:HUgY-14-721000:jKWyjq-6bPZw-14-870000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-174 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:47:12,994 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 38.5 2021-05-17 12:47:18,098 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 42.47 2021-05-17 12:47:23,176 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 47.74 2021-05-17 12:47:28,253 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 50.3 2021-05-17 12:47:33,339 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 52.91 2021-05-17 12:47:38,440 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 57.62 2021-05-17 12:47:43,528 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 62.16 2021-05-17 12:47:48,628 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 65.83 2021-05-17 12:47:53,713 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 67.75 2021-05-17 12:47:58,549 | test | INFO | pool-28-thread-2 | [table_view:display:72] Keys failed in default:_default:h0sisGTsTEYZHtrIpfq-33-633000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:47:58,838 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 70.33 2021-05-17 12:47:59,555 | test | INFO | pool-28-thread-12 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:00,865 | test | INFO | pool-28-thread-3 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-649 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-686 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-581 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:02,460 | test | INFO | pool-28-thread-11 | [table_view:display:72] Keys failed in default:scope1:GdJg9iwJNvsRzIb-33-637000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1388 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:02,983 | test | INFO | pool-28-thread-24 | [table_view:display:72] Keys failed in default:_default:329gtp8IjxUG%b4NT-33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-649 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-686 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-581 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:03,071 | test | INFO | pool-28-thread-19 | [table_view:display:72] Keys failed in default:_default:O5LYZTM5tQU-33-635000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-716 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-857 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-753 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-875 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:03,473 | test | INFO | pool-28-thread-27 | [table_view:display:72] Keys failed in default:scope1:3hc71hAH65T0nQ-s-33-636000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1040 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:04,002 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 73.26 2021-05-17 12:48:04,436 | test | INFO | pool-28-thread-6 | [table_view:display:72] Keys failed in default:scope1:GdJg9iwJNvsRzIb-33-637000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-948 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-902 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:06,773 | test | INFO | pool-28-thread-21 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:Z-33-642000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1015 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-993 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1040 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:08,517 | test | INFO | pool-28-thread-7 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-689 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-512 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:08,953 | test | INFO | pool-28-thread-9 | [table_view:display:72] Keys failed in default:_default:NyI0jF6iLVweK-33-633000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1281 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1302 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1347 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1348 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1229 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1311 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:08,953 | test | INFO | pool-28-thread-29 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:0U8_30xn5G7Y4vUR9L-33-642000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-993 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1040 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1164 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1121 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:09,099 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 76.35 2021-05-17 12:48:13,042 | test | INFO | pool-28-thread-2 | [table_view:display:72] Keys failed in default:_default:h0sisGTsTEYZHtrIpfq-33-633000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1281 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1347 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1229 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1253 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1311 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1245 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:14,203 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.43 2021-05-17 12:48:16,217 | test | INFO | pool-28-thread-12 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1281 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1347 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1249 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1253 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1354 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1200 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1311 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1245 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:16,819 | test | INFO | pool-28-thread-11 | [table_view:display:72] Keys failed in default:scope1:K-33-636000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-989 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1020 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1174 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1164 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:18,674 | test | INFO | pool-28-thread-10 | [table_view:display:72] Keys failed in default:scope1:-uq583dY-33-638000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-989 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-980 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1173 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1020 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1174 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:19,289 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 82.21 2021-05-17 12:48:20,677 | test | INFO | pool-28-thread-3 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-819 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-836 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-803 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-869 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-715 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-801 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-810 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-773 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:24,381 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 84.8 2021-05-17 12:48:25,447 | test | INFO | pool-28-thread-24 | [table_view:display:72] Keys failed in default:_default:329gtp8IjxUG%b4NT-33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-727 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-768 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-833 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-855 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-762 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-873 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:25,706 | test | INFO | pool-28-thread-13 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1435 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:26,628 | test | INFO | pool-28-thread-19 | [table_view:display:72] Keys failed in default:_default:O5LYZTM5tQU-33-635000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-935 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-969 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-933 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-950 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:26,691 | test | INFO | pool-28-thread-27 | [table_view:display:72] Keys failed in default:scope1:3hc71hAH65T0nQ-s-33-636000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1323 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1279 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1227 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1304 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1316 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1294 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1340 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1286 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:28,674 | test | INFO | pool-28-thread-7 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-727 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-768 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-833 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-855 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-710 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-762 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-760 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-750 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:29,463 | test | INFO | pool-28-thread-12 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1433 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:29,473 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 88.33 2021-05-17 12:48:30,311 | test | INFO | pool-28-thread-29 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:0U8_30xn5G7Y4vUR9L-33-642000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1279 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1316 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1286 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1341 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1353 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1365 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:32,255 | test | INFO | pool-28-thread-21 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:Z-33-642000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1378 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1246 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:33,744 | test | INFO | pool-28-thread-10 | [table_view:display:72] Keys failed in default:scope1:-uq583dY-33-638000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1378 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1246 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1198 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1187 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:33,971 | test | INFO | pool-28-thread-6 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:YtPS9H0zeS9RxLGa-33-646000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1136 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1116 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1017 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1128 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1086 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1053 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1075 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1010 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1153 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1176 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1135 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:34,549 | test | INFO | pool-28-thread-11 | [table_view:display:72] Keys failed in default:scope1:K-33-636000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1180 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1378 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1246 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1198 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1187 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:34,588 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 91.03 2021-05-17 12:48:34,832 | test | INFO | pool-28-thread-3 | [table_view:display:72] Keys failed in default:_default:ugKZ1CmLQeJDqs--33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-914 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-932 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:38,634 | test | INFO | pool-28-thread-9 | [table_view:display:72] Keys failed in default:_default:NyI0jF6iLVweK-33-633000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1402 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:39,703 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 93.96 2021-05-17 12:48:39,990 | test | INFO | pool-28-thread-13 | [table_view:display:72] Keys failed in default:_default:feZ_m0CXNT7954n-33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1017 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1007 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1018 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1057 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1003 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:40,003 | test | INFO | pool-28-thread-19 | [table_view:display:72] Keys failed in default:_default:VTKc-33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1017 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1007 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1018 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1008 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1057 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1003 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:44,799 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 96.35 2021-05-17 12:48:46,125 | test | INFO | pool-28-thread-29 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:0U8_30xn5G7Y4vUR9L-33-642000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1404 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:48,819 | test | INFO | pool-28-thread-10 | [table_view:display:72] Keys failed in default:scope1:-uq583dY-33-638000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1405 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:48,821 | test | INFO | pool-28-thread-21 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:Z-33-642000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1405 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:49,921 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: running, progress: 98.8 2021-05-17 12:48:49,954 | test | INFO | pool-28-thread-7 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-946 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-903 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-955 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-922 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-967 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-910 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:51,601 | test | INFO | pool-28-thread-14 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3209 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:51,601 | test | INFO | pool-28-thread-14 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3209 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:52,311 | test | INFO | pool-28-thread-11 | [table_view:display:72] Keys failed in default:scope1:K-33-636000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1405 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1428 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:54,937 | test | INFO | pool-28-thread-12 | [table_view:display:72] Keys failed in default:_default:VTKc-33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-616 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-604 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-657 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-500 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-577 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-556 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-545 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-652 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-642 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-664 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-650 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-693 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:55,456 | test | INFO | pool-28-thread-6 | [table_view:display:72] Keys failed in default:8ZB3TDyyZZ-33-644000:YtPS9H0zeS9RxLGa-33-646000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1181 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1182 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1349 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1250 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1330 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1374 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1264 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1221 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1331 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1375 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1277 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:56,336 | test | INFO | pool-28-thread-24 | [table_view:display:72] Keys failed in default:_default:329gtp8IjxUG%b4NT-33-634000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-903 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-922 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-934 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-964 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-971 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-972 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:57,529 | test | INFO | pool-28-thread-13 | [table_view:display:72] Keys failed in default:_default:feZ_m0CXNT7954n-33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1182 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1338 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1349 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1195 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1250 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1240 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1264 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1221 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1331 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1375 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1277 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:48:58,944 | test | INFO | pool-28-thread-27 | [table_view:display:72] Keys failed in default:scope1:3hc71hAH65T0nQ-s-33-636000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1385 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:49:00,043 | test | INFO | pool-28-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 12:49:00,200 | test | INFO | pool-28-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 169.51699996 sec 2021-05-17 12:49:01,253 | test | INFO | pool-28-thread-21 | [table_view:display:72] Keys failed in default:KpvFQU9zguHVjPe5KU-33-641000:WLIkqC-33-643000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1117 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1108 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1152 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1098 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1087 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:49:02,375 | test | INFO | pool-28-thread-19 | [table_view:display:72] Keys failed in default:_default:VTKc-33-634000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1182 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1338 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1349 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1250 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1374 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1264 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1221 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1331 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1375 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1277 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 12:53:28,891 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 12:53:28,891 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 12:53:28,894 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 4 docs from 980 to 1440 2021-05-17 12:53:28,895 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-17 12:53:28,897 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 12:53:28,897 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 12:56:25,924 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 16 docs from 980 to 1440 2021-05-17 12:56:25,924 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 17 docs from 980 to 1440 2021-05-17 12:56:25,926 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 22 docs from 980 to 1440 2021-05-17 12:56:25,926 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 23 docs from 500 to 980 2021-05-17 12:56:25,927 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 24 docs from 500 to 980 2021-05-17 12:56:25,927 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 23 docs from 980 to 1440 2021-05-17 12:56:25,927 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 20 docs from 980 to 1440 2021-05-17 12:56:25,927 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-17 12:56:25,928 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 1 docs from 2900 to 3284 2021-05-17 12:56:25,928 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 26 docs from 980 to 1440 2021-05-17 12:56:25,930 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 25 docs from 980 to 1440 2021-05-17 12:56:25,930 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 25 docs from 980 to 1440 2021-05-17 12:56:25,930 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 28 docs from 980 to 1440 2021-05-17 12:56:25,930 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 20 docs from 980 to 1440 2021-05-17 12:56:25,931 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 19 docs from 980 to 1440 2021-05-17 12:56:25,931 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 15 docs from 500 to 980 2021-05-17 12:56:25,933 | test | ERROR | pool-28-thread-25 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 13:03:28,809 | test | INFO | pool-28-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.............................................................................***X | 709.229 | | 360.357 | ......................................X | 360.357 | | 231.230 | ........................****........................***X | 451.019 | | 284.723 | ..............................X | 284.723 | | 246.008 | .........................X | 246.008 | | 246.008 | .........................*X | 246.008 | | 246.008 | .........................*..........................................................................***X | 915.125 | | 655.829 | ......................................................................X | 655.829 | | 354.372 | .....................................***X | 354.372 | | 256.457 | ...........................*****X | 256.457 | | 84.292 | ........X | 84.292 | | 84.292 | ........**............*...............***X | 333.360 | | 314.625 | .................................*****...X | 340.903 | | 340.397 | ....................................***X | 340.397 | | 291.237 | ..............................X | 291.237 | | 176.246 | ..................****X | 176.246 | | 134.280 | .............****.*.***X | 153.139 | | 110.306 | ...........*****.......******.....................................................*****X | 666.606 | | 263.611 | ...........................*****X | 263.611 | | 224.078 | .......................X | 224.078 | | 211.231 | ......................***X | 211.231 | | 210.821 | ......................X | 210.821 | | 179.456 | ..................******.........****X | 262.129 | | 176.909 | ..................X | 176.909 | | 157.403 | ................*****..............*****X | 286.791 | | 269.291 | ............................X | 269.291 | | 247.474 | ..........................***X | 247.474 | | 246.214 | .........................X | 246.214 | | 240.621 | .........................*****X | 240.621 | | 153.032 | ...............*****.......*****X | 214.437 | | 157.821 | ................******.......................****X | 367.221 | | 340.524 | ....................................X | 340.524 | | 265.524 | ............................*****..*****X | 284.439 | | 183.933 | ...................******..******.....*****X | 251.781 | | 182.889 | ..................******...............*****X | 318.417 | | 134.355 | .............******..*****....*...............****X | 324.919 | | 269.395 | ............................**X | 269.395 | | 269.395 | ............................**X | 269.395 | | 266.702 | ............................X | 266.702 | | 251.041 | ..........................**X | 251.041 | | 251.041 | ..........................**.*......****X | 312.402 | | 277.494 | .............................X | 277.494 | | 168.431 | .................****X | 168.431 | | 164.045 | ................X | 164.045 | | 130.671 | .............*****...................****X | 304.729 | | 270.217 | ............................X | 270.217 | | 197.415 | ....................****X | 197.415 | | 154.265 | ...............******..*****.*****X | 182.666 | | 175.945 | ..................************..*****X | 199.884 | | 183.043 | ...................X | 183.043 | | 183.043 | ...................**X | 183.043 | | 183.043 | ...................********...******........****X | 291.925 | | 270.628 | ............................*****X | 273.020 | | 246.939 | .........................X | 246.939 | | 229.641 | ........................***X | 229.641 | | 217.248 | ......................*...*****X | 245.368 | | 236.830 | ........................****X | 236.830 | | 220.573 | .......................*****X | 228.555 | | 225.250 | .......................*****........*...****X | 328.480 | | 306.250 | ................................*****X | 306.250 | | 209.190 | .....................*****...........******..................**X | 472.822 | | 472.822 | ..................................................*X | 472.822 | | 434.148 | ..............................................*****X | 434.148 | | 369.533 | .......................................X | 369.533 | | 360.111 | ......................................**X | 360.111 | | 360.111 | ......................................*X | 360.111 | | 325.859 | ..................................******..............*X | 456.410 | | 456.410 | ................................................**X | 456.410 | | 456.410 | ................................................X | 456.410 | | 303.730 | ................................*****X | 303.730 | | 187.047 | ...................******.*****X | 195.317 | | 153.947 | ...............******....*****........*.........X | 347.411 | | 347.411 | ....................................****.....*.**********X | 395.794 | | 320.585 | ..................................****X | 320.585 | | 284.101 | ..............................X | 284.101 | | 183.074 | ...................*****.*..*****...............*****.*..****X | 376.554 | | 228.635 | .......................******.*****..*.....*****..........*****X | 384.437 | | 283.636 | .............................*****X | 283.636 | | 216.732 | ......................******.******......****X | 280.402 | | 258.089 | ...........................X | 258.089 | | 214.156 | ......................*****...X | 241.895 | | 238.988 | .........................****X | 238.988 | | 217.613 | ......................*.*****...*...*****......******..X | 355.525 | | 353.212 | .....................................****X | 353.212 | | 248.996 | ..........................******....******........******.......................*..........****X | 660.998 | | 660.020 | .......................................................................X | 660.020 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:03:30,249 | test | INFO | pool-28-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********X | 0.200 | | 0.200 | ****X | 0.200 | | 0.000 | ************X | 0.426 | | 0.226 | X | 0.226 | | 0.000 | *******************................*****.............****X | 51.566 | | 41.960 | .......................*****....*..***X | 52.250 | | 42.646 | .......................X | 42.646 | | 37.366 | ....................****..****X | 40.663 | | 40.654 | ......................******.......................................................................*****X | 163.286 | | 75.741 | ..........................................****X | 75.741 | | 72.832 | .........................................X | 72.832 | | 69.247 | .......................................****X | 69.247 | | 61.546 | ..................................******.......******.............*****......*...*****................................*****X | 166.412 | | 53.766 | ..............................******..............................................................*****X | 161.223 | | 59.687 | .................................*****X | 59.687 | | 58.468 | ................................******..*****..........*****X | 77.845 | | 52.691 | .............................******.............*****X | 75.670 | | 58.070 | ................................****X | 58.070 | | 53.372 | .............................*...****X | 57.670 | | 53.069 | .............................X | 53.069 | | 48.770 | ...........................****X | 48.770 | | 21.195 | ...........******..................*****X | 52.580 | | 47.587 | ..........................*****X | 47.587 | | 44.296 | ........................******....*****X | 51.580 | | 47.888 | ..........................*****X | 47.888 | | 45.786 | .........................******....*****X | 53.178 | | 42.190 | .......................****X | 42.190 | | 41.588 | .......................*....****X | 49.687 | | 47.884 | ..........................*..******.*****.....................................................................X | 172.986 | | 172.986 | ..................................................................................................***X | 172.986 | | 172.986 | ..................................................................................................X | 172.986 | | 58.788 | ................................******........*****X | 71.978 | | 50.391 | ............................X | 50.391 | | 43.187 | .......................***X | 43.187 | | 41.287 | ......................*....****X | 47.497 | | 43.895 | ........................*...****X | 49.583 | | 48.183 | ..........................X | 48.183 | | 39.689 | .....................*****.......*................****X | 79.390 | | 65.684 | ....................................******..............*****X | 88.539 | | 77.515 | ...........................................*****...*......****X | 91.791 | | 90.689 | ...................................................X | 90.689 | | 76.678 | ...........................................*****..................****X | 107.286 | | 89.884 | ..................................................X | 89.884 | | 77.583 | ...........................................**********X | 77.794 | | 63.590 | ...................................******......*****X | 73.096 | | 7.125 | ...*****X | 7.125 | | 0.400 | ***********.........................................................*****X | 99.753 | | 0.400 | ******.******...................*****..........*..*****..................*......****X | 97.320 | | 82.112 | ..............................................******.*****X | 84.515 | | 22.997 | ............****X | 22.997 | | 11.901 | .....X | 11.901 | | 0.800 | ****X | 0.800 | | 0.200 | *************.**********X | 3.399 | | 2.099 | X | 2.099 | | 1.800 | *********X | 3.000 | | 0.600 | ******.*****X | 1.800 | | 1.601 | ******.*****.......................*.........*****.........................*****X | 101.318 | | 99.920 | ........................................................X | 99.920 | | 99.513 | ........................................................****X | 99.513 | | 80.008 | .............................................X | 80.008 | | 29.397 | ...............****X | 29.397 | | 25.897 | .............X | 25.897 | | 8.509 | ...*****.......*......................****X | 58.379 | | 45.982 | .........................X | 45.982 | | 36.916 | ....................****X | 36.916 | | 21.615 | ...........X | 21.615 | | 3.600 | .****X | 3.600 | | 1.601 | X | 1.601 | | 1.200 | *****..............*......****..........*...............................****X | 107.097 | | 96.493 | ......................................................X | 96.493 | | 65.707 | ....................................****X | 65.707 | | 48.625 | ...........................X | 48.625 | | 43.306 | ........................****X | 43.306 | | 13.419 | ......X | 13.419 | | 3.603 | .****X | 3.603 | | 0.200 | *X | 0.200 | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:03:31,654 | test | INFO | pool-28-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********X | 1.000 | | 0.000 | ****......*****.*****.............*****...***X | 64.294 | | 52.242 | .................****X | 52.242 | | 23.467 | .......****.........................*****..............................****X | 176.146 | | 89.739 | ...............................****X | 89.739 | | 87.621 | ..............................*****...........*....***X | 129.475 | | 78.823 | ...........................*****.....*****.*****..................................................................*****X | 278.489 | | 91.436 | ...............................******.........*****X | 116.892 | | 86.135 | .............................*****X | 86.135 | | 83.946 | .............................******..*****....*****X | 101.621 | | 82.165 | ............................******......****X | 99.627 | | 98.677 | ..................................*****X | 99.745 | | 95.851 | .................................X | 95.851 | | 67.984 | .......................*****..*........*****.X | 100.253 | | 96.556 | .................................****X | 96.556 | | 86.238 | .............................******.X | 87.176 | | 79.090 | ...........................*****...*****X | 88.260 | | 73.364 | .........................******.***********.........................................*****X | 191.151 | | 138.568 | ................................................X | 138.568 | | 88.268 | ..............................*****...*****X | 95.071 | | 85.781 | .............................*****..*.*****.......*****X | 113.870 | | 103.562 | ....................................*..*****...*****X | 118.950 | | 116.775 | ........................................*****X | 116.775 | | 83.179 | ............................******....*****X | 93.958 | | 92.181 | ................................******...........****X | 123.391 | | 105.695 | ....................................X | 105.695 | | 72.783 | .........................*****................*****X | 119.365 | | 94.480 | ................................*..***X | 97.768 | | 94.867 | .................................X | 94.867 | | 72.193 | ........................*****....*........****X | 103.167 | | 89.176 | ...............................*****X | 89.176 | | 1.800 | *****X | 1.800 | | 0.400 | ****************X | 1.100 | | 0.800 | **********X | 1.800 | | 1.400 | X | 1.400 | | 1.200 | ****X | 1.200 | | 0.800 | *****X | 0.800 | | 0.200 | ***********X | 1.200 | | 0.600 | **********X | 2.026 | | 0.801 | ***********...........*....****X | 41.916 | | 40.784 | .............*.****X | 41.950 | | 13.097 | ...X | 13.097 | | 2.798 | ****X | 2.798 | | 1.199 | ******.X | 2.999 | | 2.999 | ****X | 2.999 | | 0.599 | X | 0.599 | | 0.000 | **********************X | 2.600 | | 1.500 | X | 1.500 | | 1.200 | ***********........******...................****X | 77.426 | | 77.421 | ..........................X | 77.421 | | 71.612 | ........................****X | 71.612 | | 15.094 | ....****X | 15.094 | | 14.494 | ....X | 14.494 | | 12.513 | ...*****......*.......................****X | 92.416 | | 88.112 | ..............................X | 88.112 | | 75.017 | .........................****X | 75.017 | | 63.319 | .....................X | 63.319 | | 23.502 | .......****X | 23.502 | | 2.901 | *****X | 2.901 | | 1.400 | X | 1.400 | | 0.900 | *****X | 1.400 | | 1.100 | ***********.....*...*****.............****X | 60.512 | | 45.803 | ...............X | 45.803 | | 3.600 | ****X | 3.600 | | 3.500 | X | 3.500 | | 2.001 | ******.................*****X | 48.704 | | 33.996 | ...........****X | 33.996 | | 24.697 | .......X | 24.697 | | 4.102 | ****X | 4.102 | | 3.902 | X | 3.902 | | 2.301 | *****.*.****X | 5.702 | | 3.003 | X | 3.003 | | 2.602 | *****.*****X | 4.802 | | 2.103 | X | 2.103 | | 0.601 | *X | 0.601 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:03:31,700 | test | WARNING | Thread-92676 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8ZB3TDyyZZ-33-644000:YtPS9H0zeS9RxLGa-33-646000 update failed keys from task: [u'test_collections-1349', u'test_collections-1264', u'test_collections-1221', u'test_collections-1181', u'test_collections-1182', u'test_collections-1136', u'test_collections-1016', u'test_collections-1138', u'test_collections-1017', u'test_collections-1250', u'test_collections-1053', u'test_collections-1010', u'test_collections-1176', u'test_collections-1055', u'test_collections-1330', u'test_collections-1374', u'test_collections-1331', u'test_collections-1375', u'test_collections-1135', u'test_collections-1368', u'test_collections-1128', u'test_collections-1008', u'test_collections-1086', u'test_collections-1116', u'test_collections-1075', u'test_collections-1153', u'test_collections-1155', u'test_collections-1277'] 2021-05-17 13:03:31,700 | test | WARNING | Thread-92675 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 update failed keys from task: [u'test_collections-1412', u'test_collections-1401', u'test_collections-1435', u'test_collections-1387'] 2021-05-17 13:03:31,707 | test | WARNING | Thread-92705 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:h0sisGTsTEYZHtrIpfq-33-633000 update failed keys from task: [u'test_collections-1347', u'test_collections-1204', u'test_collections-1249', u'test_collections-1229', u'test_collections-1360', u'test_collections-1320', u'test_collections-1245', u'test_collections-1223', u'test_collections-1281', u'test_collections-1217', u'test_collections-1172', u'test_collections-1230', u'test_collections-1373', u'test_collections-1252', u'test_collections-1253', u'test_collections-1013', u'test_collections-1311'] 2021-05-17 13:03:31,709 | test | WARNING | Thread-92709 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ugKZ1CmLQeJDqs--33-634000 update failed keys from task: [u'test_collections-1347', u'test_collections-1382', u'test_collections-1223', u'test_collections-1015', u'test_collections-1414', u'test_collections-1217', u'test_collections-1172', u'test_collections-1373', u'test_collections-1252', u'test_collections-1253', u'test_collections-1013', u'test_collections-1204', u'test_collections-1249', u'test_collections-1360', u'test_collections-1320', u'test_collections-1200', u'test_collections-1245', u'test_collections-1281', u'test_collections-1433', u'test_collections-1230', u'test_collections-1354', u'test_collections-1311'] 2021-05-17 13:03:31,710 | test | WARNING | Thread-92710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:VTKc-33-634000 update failed keys from task: [u'test_collections-1368', u'test_collections-1007', u'test_collections-1008', u'test_collections-1349', u'test_collections-1264', u'test_collections-1045', u'test_collections-1221', u'test_collections-1189', u'test_collections-1003', u'test_collections-1182', u'test_collections-1017', u'test_collections-1018', u'test_collections-1338', u'test_collections-1250', u'test_collections-1374', u'test_collections-1331', u'test_collections-1375', u'test_collections-1057', u'test_collections-1277'] 2021-05-17 13:03:31,710 | test | WARNING | Thread-92714 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:feZ_m0CXNT7954n-33-634000 update failed keys from task: [u'test_collections-1368', u'test_collections-1007', u'test_collections-1008', u'test_collections-1349', u'test_collections-1240', u'test_collections-1264', u'test_collections-1045', u'test_collections-1221', u'test_collections-1189', u'test_collections-1003', u'test_collections-1182', u'test_collections-1017', u'test_collections-1018', u'test_collections-1338', u'test_collections-1195', u'test_collections-1250', u'test_collections-1331', u'test_collections-1375', u'test_collections-1057', u'test_collections-1277'] 2021-05-17 13:03:31,713 | test | WARNING | Thread-92727 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:NyI0jF6iLVweK-33-633000 update failed keys from task: [u'test_collections-1292', u'test_collections-1302', u'test_collections-1325', u'test_collections-1347', u'test_collections-1402', u'test_collections-1348', u'test_collections-1249', u'test_collections-1229', u'test_collections-1360', u'test_collections-1223', u'test_collections-1281', u'test_collections-1217', u'test_collections-1230', u'test_collections-1373', u'test_collections-1252', u'test_collections-1311'] 2021-05-17 13:03:31,710 | test | WARNING | Thread-92716 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:O5LYZTM5tQU-33-635000 delete failed keys from task: [u'test_collections-969', u'test_collections-780', u'test_collections-935', u'test_collections-716', u'test_collections-933', u'test_collections-857', u'test_collections-954', u'test_collections-852', u'test_collections-753', u'test_collections-731', u'test_collections-875', u'test_collections-950', u'test_collections-892'] 2021-05-17 13:03:31,806 | test | WARNING | Thread-93066 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:HUgY-14-721000:jKWyjq-6bPZw-14-870000 create failed keys from task: [u'test_collections-174'] 2021-05-17 13:03:31,825 | test | WARNING | Thread-93135 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:nojN3VMu3H-15-414000:JOiuiPqs5-15-525000 create failed keys from task: [u'test_collections-174'] 2021-05-17 13:03:31,858 | test | WARNING | Thread-92814 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:-uq583dY-33-638000 update failed keys from task: [u'test_collections-1428', u'test_collections-1020', u'test_collections-1262', u'test_collections-1186', u'test_collections-1187', u'test_collections-1180', u'test_collections-1059', u'test_collections-1378', u'test_collections-1173', u'test_collections-1174', u'test_collections-1212', u'test_collections-988', u'test_collections-989', u'test_collections-982', u'test_collections-1246', u'test_collections-980', u'test_collections-1405', u'test_collections-1161', u'test_collections-1164', u'test_collections-1244', u'test_collections-1201', u'test_collections-1366', u'test_collections-1031', u'test_collections-1351', u'test_collections-1198'] 2021-05-17 13:03:31,874 | test | WARNING | Thread-92735 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:329gtp8IjxUG%b4NT-33-634000 delete failed keys from task: [u'test_collections-649', u'test_collections-727', u'test_collections-768', u'test_collections-922', u'test_collections-964', u'test_collections-762', u'test_collections-686', u'test_collections-760', u'test_collections-937', u'test_collections-934', u'test_collections-535', u'test_collections-855', u'test_collections-610', u'test_collections-971', u'test_collections-972', u'test_collections-749', u'test_collections-903', u'test_collections-541', u'test_collections-581', u'test_collections-518', u'test_collections-558', u'test_collections-833', u'test_collections-873', u'test_collections-793'] 2021-05-17 13:03:31,875 | test | WARNING | Thread-92675 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 delete failed keys from task: [u'test_collections-946', u'test_collections-727', u'test_collections-903', u'test_collections-768', u'test_collections-922', u'test_collections-967', u'test_collections-689', u'test_collections-762', u'test_collections-760', u'test_collections-541', u'test_collections-955', u'test_collections-833', u'test_collections-855', u'test_collections-910', u'test_collections-610', u'test_collections-512', u'test_collections-710', u'test_collections-750'] 2021-05-17 13:03:31,875 | test | WARNING | Thread-92767 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KpvFQU9zguHVjPe5KU-33-641000:0U8_30xn5G7Y4vUR9L-33-642000 update failed keys from task: [u'test_collections-1428', u'test_collections-1262', u'test_collections-1186', u'test_collections-1143', u'test_collections-1341', u'test_collections-1146', u'test_collections-1180', u'test_collections-1056', u'test_collections-988', u'test_collections-1404', u'test_collections-1040', u'test_collections-1284', u'test_collections-1164', u'test_collections-1121', u'test_collections-1286', u'test_collections-1365', u'test_collections-1124', u'test_collections-1366', u'test_collections-1080', u'test_collections-993', u'test_collections-1279', u'test_collections-1314', u'test_collections-1316', u'test_collections-1110', u'test_collections-1275', u'test_collections-1353'] 2021-05-17 13:03:31,915 | test | WARNING | Thread-92675 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8ZB3TDyyZZ-33-644000:mvGxsh5fxVvl-33-646000 create failed keys from task: [u'test_collections-3209'] 2021-05-17 13:03:31,944 | test | WARNING | Thread-92799 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:3hc71hAH65T0nQ-s-33-636000 update failed keys from task: [u'test_collections-1323', u'test_collections-1227', u'test_collections-1304', u'test_collections-1040', u'test_collections-1284', u'test_collections-1340', u'test_collections-1022', u'test_collections-1286', u'test_collections-1341', u'test_collections-1385', u'test_collections-1080', u'test_collections-1015', u'test_collections-1279', u'test_collections-1379', u'test_collections-1314', u'test_collections-1316', u'test_collections-1172', u'test_collections-1294', u'test_collections-1110', u'test_collections-1013'] 2021-05-17 13:03:31,983 | test | WARNING | Thread-92800 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:GdJg9iwJNvsRzIb-33-637000 update failed keys from task: [u'test_collections-1388'] 2021-05-17 13:03:32,101 | test | WARNING | Thread-92800 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:GdJg9iwJNvsRzIb-33-637000 delete failed keys from task: [u'test_collections-948', u'test_collections-902', u'test_collections-925', u'test_collections-951', u'test_collections-960'] 2021-05-17 13:03:32,122 | test | WARNING | Thread-92773 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KpvFQU9zguHVjPe5KU-33-641000:Z-33-642000 update failed keys from task: [u'test_collections-1428', u'test_collections-1262', u'test_collections-1186', u'test_collections-1143', u'test_collections-1146', u'test_collections-1180', u'test_collections-1015', u'test_collections-1378', u'test_collections-1056', u'test_collections-1013', u'test_collections-1212', u'test_collections-988', u'test_collections-1246', u'test_collections-1405', u'test_collections-1040', u'test_collections-1244', u'test_collections-1124', u'test_collections-1201', u'test_collections-1366', u'test_collections-1080', u'test_collections-993', u'test_collections-1351', u'test_collections-1110'] 2021-05-17 13:03:33,174 | test | WARNING | Thread-92807 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:K-33-636000 update failed keys from task: [u'test_collections-1428', u'test_collections-1020', u'test_collections-1262', u'test_collections-1186', u'test_collections-1187', u'test_collections-1180', u'test_collections-1059', u'test_collections-1378', u'test_collections-1174', u'test_collections-1212', u'test_collections-988', u'test_collections-989', u'test_collections-982', u'test_collections-1246', u'test_collections-980', u'test_collections-1405', u'test_collections-1161', u'test_collections-1164', u'test_collections-1244', u'test_collections-1201', u'test_collections-1366', u'test_collections-991', u'test_collections-1031', u'test_collections-1351', u'test_collections-1198'] 2021-05-17 13:03:33,348 | test | WARNING | Thread-92710 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:VTKc-33-634000 delete failed keys from task: [u'test_collections-604', u'test_collections-500', u'test_collections-545', u'test_collections-642', u'test_collections-664', u'test_collections-616', u'test_collections-657', u'test_collections-533', u'test_collections-577', u'test_collections-556', u'test_collections-652', u'test_collections-631', u'test_collections-653', u'test_collections-650', u'test_collections-693'] 2021-05-17 13:03:33,349 | test | WARNING | Thread-92752 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KpvFQU9zguHVjPe5KU-33-641000:WLIkqC-33-643000 update failed keys from task: [u'test_collections-1117', u'test_collections-1019', u'test_collections-1108', u'test_collections-1152', u'test_collections-1098', u'test_collections-1087'] 2021-05-17 13:03:33,588 | test | WARNING | Thread-92709 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:ugKZ1CmLQeJDqs--33-634000 delete failed keys from task: [u'test_collections-649', u'test_collections-803', u'test_collections-801', u'test_collections-767', u'test_collections-686', u'test_collections-818', u'test_collections-819', u'test_collections-535', u'test_collections-810', u'test_collections-932', u'test_collections-930', u'test_collections-773', u'test_collections-869', u'test_collections-541', u'test_collections-783', u'test_collections-581', u'test_collections-917', u'test_collections-759', u'test_collections-836', u'test_collections-715', u'test_collections-518', u'test_collections-914', u'test_collections-558'] 2021-05-17 13:03:45,131 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 13:04:49,026 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 13:04:49,028 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 13:05:07,299 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 13:05:07,301 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 13:05:32,930 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 13:05:32,930 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 13:06:28,834 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1210000515 seconds to complete 2021-05-17 13:06:28,848 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3558.178s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 26 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_26 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_27 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_27', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 27, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 13:06:38,200 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #27 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 13:06:38,885 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #27 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 13:06:39,332 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xf79c62 type=client open_count=1 channel=[id: 0x95e7c230, 0.0.0.0/0.0.0.0:36597] timeout=300.0>) 2021-05-17 13:07:00,092 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xf79c63 type=client open_count=1 channel=[id: 0x8bf936a2, 0.0.0.0/0.0.0.0:60447] timeout=300.0>) 2021-05-17 13:07:02,109 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xf79c64 type=client open_count=1 channel=[id: 0x8d9b54fd, 0.0.0.0/0.0.0.0:44268] timeout=300.0>) 2021-05-17 13:07:04,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf79c65 type=client open_count=1 channel=[id: 0x4f702f1d, 0.0.0.0/0.0.0.0:45071] timeout=300.0>) 2021-05-17 13:07:27,953 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf79c66 type=client open_count=1 channel=[id: 0x8c72cfca, 0.0.0.0/0.0.0.0:47265] timeout=300.0>) 2021-05-17 13:07:29,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf79c67 type=client open_count=1 channel=[id: 0xcab1c685, 0.0.0.0/0.0.0.0:40667] timeout=300.0>) 2021-05-17 13:07:31,987 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xf79c68 type=client open_count=1 channel=[id: 0x3a03ad65, 0.0.0.0/0.0.0.0:44045] timeout=300.0>) 2021-05-17 13:07:56,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xf79c69 type=client open_count=1 channel=[id: 0xa9598eec, 0.0.0.0/0.0.0.0:60353] timeout=300.0>) 2021-05-17 13:07:58,131 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xf79c6a type=client open_count=1 channel=[id: 0x30fa1a2a, 0.0.0.0/0.0.0.0:40246] timeout=300.0>) 2021-05-17 13:08:00,148 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xf79c6b type=client open_count=1 channel=[id: 0x38fe9868, 0.0.0.0/0.0.0.0:59889] timeout=300.0>) 2021-05-17 13:08:23,948 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xf79c6c type=client open_count=1 channel=[id: 0x3780b8d3, 0.0.0.0/0.0.0.0:58652] timeout=300.0>) 2021-05-17 13:08:25,969 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xf79c6d type=client open_count=1 channel=[id: 0xeb2b605d, 0.0.0.0/0.0.0.0:39366] timeout=300.0>) 2021-05-17 13:08:27,986 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xf79c6e type=client open_count=1 channel=[id: 0x877ee5c8, 0.0.0.0/0.0.0.0:42979] timeout=300.0>) 2021-05-17 13:08:52,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xf79c6f type=client open_count=1 channel=[id: 0xf22a5686, 0.0.0.0/0.0.0.0:55939] timeout=300.0>) 2021-05-17 13:08:54,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xf79c70 type=client open_count=1 channel=[id: 0x1315a7ce, 0.0.0.0/0.0.0.0:53309] timeout=300.0>) 2021-05-17 13:08:56,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 13:09:13,862 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 13:09:27,523 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 13:09:27,548 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #27 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 13:09:27,585 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #27 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 13:09:41,278 | test | INFO | pool-29-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.36155461485 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 13:09:41,302 | test | INFO | pool-29-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:09:51,374 | test | INFO | pool-29-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 13:09:51,401 | test | INFO | pool-29-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1229999065 sec 2021-05-17 13:09:51,661 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 13:10:42,908 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 13:10:43,266 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 696676520 | 76589459 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 676300304 | 103280476 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 970315256 | 45162103 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 13:10:43,267 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 13:11:30,753 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 13:11:30,760 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 13:29:01,359 | test | INFO | pool-29-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***................................*****..............*......................................................***X | 708.714 | | 623.120 | ......................................................................................X | 623.120 | | 567.621 | ...............................................................................****X | 567.621 | | 508.448 | ......................................................................****X | 508.448 | | 504.779 | ......................................................................*.****X | 511.499 | | 499.279 | .....................................................................X | 499.279 | | 481.805 | ..................................................................*****...****X | 501.604 | | 495.022 | ....................................................................X | 495.022 | | 488.890 | ...................................................................****X | 488.890 | | 472.585 | .................................................................*****.*..***X | 490.456 | | 471.996 | .................................................................X | 471.996 | | 463.167 | ................................................................****X | 463.167 | | 405.502 | ........................................................*****.....*****X | 443.169 | | 300.212 | .........................................*****................******.........******......****X | 518.423 | | 483.957 | ...................................................................X | 483.957 | | 408.746 | ........................................................*****.............*****X | 500.058 | | 460.818 | ................................................................*****X | 460.818 | | 415.394 | .........................................................****X | 415.394 | | 407.331 | ........................................................*.*****...................*****X | 550.624 | | 513.059 | .......................................................................******...........****X | 593.353 | | 544.451 | ...........................................................................X | 544.451 | | 476.581 | ..................................................................**********X | 479.492 | | 438.806 | ............................................................****X | 438.806 | | 435.973 | ............................................................X | 435.973 | | 420.733 | ..........................................................*****.............******............****X | 597.180 | | 579.421 | ................................................................................X | 579.421 | | 530.668 | .........................................................................X | 530.668 | | 521.471 | ........................................................................****.X | 530.288 | | 526.342 | .........................................................................****X | 526.342 | | 504.201 | ......................................................................*****X | 504.201 | | 463.901 | ................................................................X | 463.901 | | 452.600 | ..............................................................****....*.*****..X | 497.579 | | 453.345 | ..............................................................*****.******....****X | 488.474 | | 479.124 | ..................................................................*.X | 485.207 | | 484.254 | ...................................................................****......*..****X | 545.065 | | 406.907 | ........................................................*****..*..........****X | 494.298 | | 455.546 | ...............................................................*****.....X | 490.528 | | 485.915 | ...................................................................*....*****..****X | 524.562 | | 520.162 | ........................................................................X | 520.162 | | 479.276 | ..................................................................****X | 479.276 | | 469.230 | .................................................................****X | 469.230 | | 456.742 | ...............................................................X | 456.742 | | 441.076 | .............................................................*..****.....X | 491.573 | | 461.661 | ................................................................****X | 461.661 | | 457.220 | ...............................................................X | 457.220 | | 440.594 | .............................................................*****............*****X | 526.187 | | 472.370 | .................................................................*****........*..****X | 545.524 | | 418.919 | ..........................................................X | 418.919 | | 388.335 | .....................................................****...*......*****....****X | 479.051 | | 469.103 | .................................................................X | 469.103 | | 469.048 | .................................................................****.....*****X | 505.429 | | 504.966 | ......................................................................*..****X | 523.815 | | 410.057 | ........................................................X | 410.057 | | 398.358 | .......................................................****............*......****X | 524.965 | | 473.895 | .................................................................*****X | 473.895 | | 413.981 | .........................................................*****........*****X | 471.985 | | 419.575 | ..........................................................******......*****...*...****X | 509.483 | | 500.174 | .....................................................................X | 500.174 | | 467.185 | ................................................................****X | 467.185 | | 409.311 | ........................................................******.........****X | 472.050 | | 421.372 | ..........................................................*****X | 421.372 | | 401.707 | .......................................................*****...............****X | 508.309 | | 503.999 | ......................................................................*.****X | 514.420 | | 434.320 | ............................................................****X | 434.320 | | 330.565 | .............................................******........*..............****X | 484.439 | | 394.161 | ......................................................*****..*......*****.....******.........****X | 546.821 | | 512.754 | .......................................................................*****X | 513.812 | | 460.747 | ................................................................******.....***********..*.....****X | 546.511 | | 424.730 | ..........................................................*****..........*.....****X | 525.645 | | 457.857 | ...............................................................******...****X | 475.640 | | 458.340 | ...............................................................X | 458.340 | | 396.994 | .......................................................*****.................*****X | 521.623 | | 451.945 | ..............................................................*****X | 451.945 | | 416.822 | .........................................................*****..........*...*****..**********X | 523.870 | | 496.379 | .....................................................................X | 496.379 | | 448.587 | ..............................................................*****.............*****X | 540.751 | | 509.190 | ......................................................................*****X | 509.190 | | 382.591 | ....................................................******............*****........*..****X | 536.541 | | 499.608 | .....................................................................X | 499.608 | | 480.233 | ..................................................................*X | 480.233 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:29:02,506 | test | INFO | pool-29-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***....................................................................................................****X | 30.002 | | 24.002 | ...............................................................................X | 24.002 | | 0.000 | ******************************************************....*****...........*****X | 4.501 | | 0.000 | ***********....*****X | 1.500 | | 0.000 | *****..............*****X | 4.499 | | 0.000 | ****************************....*.*****.*...****X | 3.000 | | 2.700 | .......X | 2.700 | | 1.500 | ...*****.....****X | 3.000 | | 2.200 | ......X | 2.200 | | 0.200 | X | 0.200 | | 0.000 | **********.....****X | 1.500 | | 0.800 | .X | 0.800 | | 0.100 | X | 0.100 | | 0.000 | ****....*****X | 1.300 | | 0.200 | *****X | 0.200 | | 0.000 | ******....****X | 1.500 | | 0.000 | ***********...........*.............****X | 7.497 | | 0.500 | X | 0.500 | | 0.000 | ****************...*.*****X | 1.500 | | 1.500 | ...****X | 1.500 | | 0.000 | *****************.........****X | 3.000 | | 1.600 | ....X | 1.600 | | 0.000 | *****.*...****X | 1.498 | | 0.000 | *****************...................*****.....*.....................****X | 13.685 | | 13.085 | ..........................................X | 13.085 | | 10.189 | ................................****X | 10.189 | | 0.500 | ******...*****X | 1.499 | | 1.499 | ...****..........****X | 4.496 | | 4.196 | ............X | 4.196 | | 0.000 | *****.....*........................****X | 8.993 | | 2.998 | ........****X | 2.998 | | 0.998 | ..X | 0.998 | | 0.000 | *****.........*...............****X | 7.495 | | 2.998 | ........*****X | 2.998 | | 0.000 | ****************..................................*****X | 10.494 | | 6.294 | ...................X | 6.294 | | 1.499 | ...*****...............****X | 5.997 | | 4.096 | ............******..................****X | 9.398 | | 5.997 | ..................****X | 5.997 | | 5.596 | .................X | 5.596 | | 0.000 | *****.......................................*****X | 11.896 | | 9.096 | .............................X | 9.096 | | 6.097 | ...................****X | 6.097 | | 4.497 | .............*****........................******..****X | 12.294 | | 11.494 | .....................................X | 11.494 | | 1.499 | ...****X | 1.499 | | 0.000 | ******..*..****X | 1.499 | | 0.000 | ***********........*****.............*.................*****.............****X | 15.396 | | 9.799 | ...............................X | 9.799 | | 9.398 | ..............................****X | 9.398 | | 4.600 | ..............*****X | 4.600 | | 2.898 | ........****X | 2.898 | | 1.400 | ...X | 1.400 | | 0.700 | .*****................*..............****X | 9.798 | | 8.999 | ............................*****X | 8.999 | | 8.399 | ..........................X | 8.399 | | 5.999 | ..................****X | 5.999 | | 0.000 | *****************..................................****X | 10.497 | | 9.797 | ...............................X | 9.797 | | 0.000 | *******************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 13:29:03,342 | test | INFO | pool-29-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***....................................................................................................****X | 84.005 | | 67.205 | ...............................................................................X | 67.205 | | 0.000 | **********************************************************************.....****X | 4.500 | | 2.100 | .X | 2.100 | | 0.000 | *****.*****X | 1.500 | | 1.500 | *****X | 1.500 | | 0.000 | *****************************.*****.............*****X | 12.001 | | 0.000 | *****....*....****X | 7.500 | | 6.000 | ......X | 6.000 | | 0.000 | *****.****X | 1.500 | | 0.800 | X | 0.800 | | 0.100 | X | 0.100 | | 0.000 | ****************.****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | *****.....*****X | 4.500 | | 0.000 | *****.******.........*.****X | 9.396 | | 9.096 | .........X | 9.096 | | 7.102 | .......****X | 7.102 | | 2.900 | ..****X | 2.900 | | 1.500 | X | 1.500 | | 0.100 | *****..........****X | 9.000 | | 0.600 | X | 0.600 | | 0.000 | *****.......*****X | 6.000 | | 0.000 | ************...****X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | ******...***X | 2.996 | | 2.200 | .X | 2.200 | | 0.200 | X | 0.200 | | 0.000 | ****************...*****.*.*****.*.***X | 5.994 | | 4.894 | ....X | 4.894 | | 4.495 | ....****X | 4.495 | | 0.000 | *****.....*.......***X | 10.490 | | 0.000 | ******.*****..*.....****X | 7.494 | | 2.998 | ..*****X | 2.998 | | 0.200 | X | 0.200 | | 0.000 | ****....*.*********X | 4.497 | | 4.397 | ....X | 4.397 | | 2.998 | ..****X | 2.998 | | 2.398 | .X | 2.398 | | 0.000 | ***********.******.*****..*****..........*****X | 11.889 | | 6.099 | ......****X | 6.099 | | 2.998 | ..****X | 2.998 | | 2.698 | ..X | 2.698 | | 1.499 | *****..****X | 2.999 | | 1.499 | *****X | 1.499 | | 0.000 | ******...........****X | 9.796 | | 3.698 | ...*****X | 3.698 | | 0.000 | *********************************..................*****X | 15.696 | | 9.798 | ..........*****X | 9.798 | | 0.000 | *****************.*****X | 1.500 | | 1.500 | ******..****X | 3.000 | | 2.999 | ..X | 2.999 | | 2.999 | ..****X | 2.999 | | 0.000 | *****************...*****X | 2.999 | | 0.000 | ********************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 13:29:03,607 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 67.518686599 | 6437847040 | 4336271360 | 2398306304 / 7511588864 | 101737 / 202639 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 69.4 | 6437847040 | 4222275584 | 2483322880 / 7511588864 | 99753 / 202049 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 63.9593993233 | 6437847040 | 4225224704 | 2472779776 / 7511588864 | 101025 / 203440 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 68.7998933387 | 6437847040 | 3824218112 | 3058667520 / 9530753024 | 100744 / 200642 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 68.5156171872 | 6437847040 | 4281307136 | 2439782400 / 7511588864 | 101335 / 200416 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 13:29:16,496 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 13:30:42,913 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164158264 | 244084121 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162046160 | 278152318 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 444015512 | 458344620 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 13:30:42,914 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #27 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 13:30:53,048 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 13:30:53,049 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 13:30:53,049 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 13:31:35,013 | test | INFO | pool-29-thread-27 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 41.7780001163 seconds to complete 2021-05-17 13:31:35,017 | test | INFO | pool-29-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 13:33:40,078 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0569999217987 seconds 2021-05-17 13:33:40,078 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 13:33:40,079 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 13:33:46,773 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 13:44:22,720 | test | INFO | pool-29-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****.......*..*****.............*.....*****...................................................*.................****X | 1001.473 | | 381.333 | ...................................******................................................................*****X | 1044.502 | | 548.139 | ...................................................****X | 548.139 | | 484.297 | .............................................X | 484.297 | | 393.175 | ....................................X | 393.175 | | 359.625 | .................................****...........******...*****X | 507.574 | | 449.203 | ..........................................*****X | 449.203 | | 268.884 | ........................*****X | 268.884 | | 244.569 | ......................*X | 244.569 | | 244.569 | ......................***X | 244.569 | | 225.551 | ....................***X | 225.551 | | 225.551 | ....................*X | 225.551 | | 146.338 | .............******.......*****X | 227.682 | | 176.499 | ...............*****.******....*...........*****.*.****X | 358.986 | | 311.367 | ............................X | 311.367 | | 295.051 | ...........................****X | 295.051 | | 268.042 | ........................X | 268.042 | | 254.411 | .......................****X | 254.411 | | 226.666 | ....................X | 226.666 | | 225.377 | ....................*****...*.*****..................................*...........****X | 740.633 | | 656.833 | .............................................................X | 656.833 | | 381.235 | ...................................****X | 381.235 | | 358.492 | .................................X | 358.492 | | 352.087 | ................................**X | 352.087 | | 352.087 | ................................**..............*.....*****........*.X | 645.526 | | 645.526 | ............................................................**X | 645.526 | | 645.526 | ............................................................X | 645.526 | | 642.054 | ............................................................*..****X | 664.099 | | 579.936 | ......................................................X | 579.936 | | 540.595 | ..................................................X | 540.595 | | 540.595 | ..................................................**X | 540.595 | | 540.595 | ..................................................X | 540.595 | | 536.512 | ..................................................X | 536.512 | | 535.098 | ..................................................***X | 535.098 | | 472.039 | ............................................X | 472.039 | | 457.556 | ..........................................*****.......*......*****......*****X | 652.996 | | 598.792 | ........................................................X | 598.792 | | 598.792 | ........................................................**X | 598.792 | | 598.792 | ........................................................*X | 600.378 | | 587.535 | .......................................................*****........*..****X | 696.891 | | 645.452 | ............................................................X | 645.452 | | 464.812 | ...........................................*****.....*****X | 514.991 | | 415.284 | ......................................******.............******.....****X | 599.769 | | 580.054 | ......................................................X | 580.054 | | 500.940 | ..............................................****X | 500.940 | | 500.640 | ..............................................X | 500.640 | | 487.607 | .............................................****X | 487.607 | | 443.907 | .........................................X | 443.907 | | 361.552 | .................................*****.........******......*****X | 513.086 | | 500.039 | ..............................................*****X | 500.039 | | 454.501 | ..........................................******.*......*****X | 524.619 | | 480.104 | ............................................****X | 480.104 | | 404.222 | .....................................******........******.X | 492.487 | | 479.172 | ............................................****X | 479.172 | | 450.270 | ..........................................******...****X | 484.778 | | 476.947 | ............................................X | 476.947 | | 469.447 | ...........................................*****..*......****X | 548.973 | | 521.131 | ................................................******.........*****X | 612.739 | | 588.788 | .......................................................****X | 588.788 | | 556.971 | ....................................................X | 556.971 | | 541.106 | ..................................................*****......*...............**X | 762.093 | +---------+-----------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 13:44:23,778 | test | INFO | pool-29-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....*...*****...............................................******.................................****X | 144.732 | | 139.120 | ..................................................................................X | 139.120 | | 134.187 | ...............................................................................*****...........*****X | 151.988 | | 127.590 | ...........................................................................******........****X | 141.101 | | 126.919 | ...........................................................................X | 126.919 | | 115.600 | ....................................................................*****..............*.......*****...*.****X | 156.707 | | 133.295 | ...............................................................................X | 133.295 | | 129.194 | ............................................................................*****.*..****X | 132.989 | | 129.292 | ............................................................................******..X | 131.878 | | 128.733 | ............................................................................*****.....*....*****..*..*****.........*****X | 166.089 | | 122.519 | ........................................................................X | 122.519 | | 109.000 | ................................................................****.....*.............*........***X | 152.400 | | 138.396 | ..................................................................................X | 138.396 | | 76.790 | .............................................***********.*......*****X | 88.891 | | 74.793 | ............................................******.......*....****X | 93.308 | | 81.486 | ................................................*****X | 81.486 | | 69.718 | ........................................*****X | 69.718 | | 21.500 | ...........****X | 21.500 | | 15.300 | ........X | 15.300 | | 2.001 | ****X | 2.001 | | 1.701 | X | 1.701 | | 1.500 | *****.*.*****.****X | 6.001 | | 3.701 | .X | 3.701 | | 2.001 | ******.****X | 3.500 | | 2.100 | X | 2.100 | | 0.700 | *****..***********X | 3.700 | | 1.900 | ******************.*****X | 4.000 | | 1.899 | ************....****X | 8.696 | | 0.400 | ******..*****X | 4.799 | | 0.200 | ******..*.*X | 6.600 | | 6.600 | ..**X | 6.600 | | 1.100 | ******.*****X | 3.200 | | 2.500 | X | 2.500 | | 2.000 | **********.*****X | 3.800 | | 1.700 | ******.*****X | 4.299 | | 2.000 | *******..*****X | 5.199 | | 1.400 | ******.****X | 2.300 | | 1.300 | ******..*****X | 4.499 | | 3.800 | .******.****X | 5.100 | | 1.700 | X | 1.700 | | 1.200 | ***X | 1.200 | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:44:24,394 | test | INFO | pool-29-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...*...*****.......................*...*****.............................................*****.......*.....*****.....*****X | 255.096 | | 214.273 | ..............................................................................*****X | 214.273 | | 210.101 | ............................................................................******.......*****X | 227.614 | | 166.579 | ............................................................*****....*...........*****......*..................****X | 269.677 | | 248.068 | ..........................................................................................X | 248.068 | | 173.688 | ...............................................................****X | 173.688 | | 170.297 | ..............................................................*.*..***X | 179.064 | | 176.573 | ................................................................*.X | 178.173 | | 176.154 | ................................................................****.............*.....*****.........*****X | 250.595 | | 235.201 | ......................................................................................*****X | 235.201 | | 203.505 | ..........................................................................*****X | 203.505 | | 200.491 | .........................................................................*****X | 200.491 | | 132.584 | ................................................X | 132.584 | | 62.794 | ......................*****...*****X | 72.606 | | 71.389 | .........................X | 71.389 | | 68.590 | ........................*****.*****X | 70.894 | | 23.902 | .......*****X | 23.902 | | 7.006 | .X | 7.006 | | 0.700 | **********X | 1.000 | | 0.400 | ******.*****X | 4.301 | | 2.400 | ****X | 2.400 | | 1.301 | X | 1.301 | | 0.600 | *****************.*****X | 4.101 | | 2.200 | ****X | 2.200 | | 1.200 | X | 1.200 | | 0.000 | ******X | 1.800 | | 1.800 | *X | 1.800 | | 1.800 | **.***********X | 4.700 | | 2.000 | *****X | 2.000 | | 1.300 | X | 1.300 | | 0.900 | *****.*****X | 4.399 | | 1.500 | ***********X | 1.899 | | 1.100 | ******..*****X | 6.499 | | 1.000 | ***********X | 1.200 | | 0.900 | ******.*****X | 5.300 | | 2.300 | X | 2.300 | | 0.200 | **********.*****X | 4.400 | | 3.000 | *****X | 3.000 | | 1.600 | ******.*****X | 3.101 | | 2.200 | ****X | 2.200 | | 1.400 | *******.****X | 3.200 | | 0.200 | ******.*****************X | 4.500 | | 1.300 | ****X | 1.300 | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:44:24,668 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 13:44:24,719 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 13:44:34,910 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 13:44:35,023 | test | INFO | pool-29-thread-26 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.415 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 34.8984115872 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 66.434440574 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 34.5316666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 27.7342578086 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 13:44:35,062 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:44:40,197 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:44:42,782 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 13:44:45,269 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:44:50,342 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:44:55,414 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:45:00,477 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 13:45:05,552 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.99 2021-05-17 13:45:06,743 | test | INFO | pool-29-thread-16 | [table_view:display:72] Keys failed in bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:07,642 | test | INFO | pool-29-thread-29 | [table_view:display:72] Keys failed in bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-254 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-265 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:10,664 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.27 2021-05-17 13:45:15,737 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 10.73 2021-05-17 13:45:16,381 | test | INFO | pool-29-thread-16 | [table_view:display:72] Keys failed in bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1507 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1588 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1458 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1469 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1547 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1492 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1582 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1497 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1542 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:20,825 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.77 2021-05-17 13:45:21,352 | test | INFO | pool-29-thread-29 | [table_view:display:72] Keys failed in bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-529 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-535 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-512 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-476 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-575 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-433 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-570 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-482 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:25,933 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.41 2021-05-17 13:45:31,042 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 23.77 2021-05-17 13:45:31,181 | test | INFO | pool-29-thread-16 | [table_view:display:72] Keys failed in bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1605 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1795 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1685 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:36,158 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 27.26 2021-05-17 13:45:41,236 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.42 2021-05-17 13:45:46,306 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 13:45:51,405 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 37.76 2021-05-17 13:45:56,017 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in bucket2:eSraAKxUCyos-43-995000:AcfTfL%lZ%4PY80BbO2_-44-117000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:45:56,533 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 42.38 2021-05-17 13:46:00,676 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in bucket2:eSraAKxUCyos-43-995000:AcfTfL%lZ%4PY80BbO2_-44-117000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1466 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1565 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:01,650 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 46.4 2021-05-17 13:46:05,005 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in bucket2:eSraAKxUCyos-43-995000:AcfTfL%lZ%4PY80BbO2_-44-117000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:06,749 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 50.92 2021-05-17 13:46:12,104 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 53.99 2021-05-17 13:46:17,224 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 57.73 2021-05-17 13:46:22,354 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 61.89 2021-05-17 13:46:27,479 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 65.48 2021-05-17 13:46:32,588 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.78 2021-05-17 13:46:37,723 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 69.36 2021-05-17 13:46:42,861 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 72.63 2021-05-17 13:46:43,157 | test | INFO | pool-29-thread-19 | [table_view:display:72] Keys failed in default:_default:wVQXxO5-J7DQAketRNzc-51-747000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:43,157 | test | INFO | pool-29-thread-16 | [table_view:display:72] Keys failed in default:_default:uxE2HFim3uKAL1py3YDB-51-747000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:43,336 | test | INFO | pool-29-thread-7 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:43,891 | test | INFO | pool-29-thread-3 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-649 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-677 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-650 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-563 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:44,154 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-526 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-402 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-487 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:44,375 | test | INFO | pool-29-thread-30 | [table_view:display:72] Keys failed in default:_default:Uff-51-748000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1161 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:45,043 | test | INFO | pool-29-thread-18 | [table_view:display:72] Keys failed in default:_default:U6bD__z-51-749000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:45,305 | test | INFO | pool-29-thread-27 | [table_view:display:72] Keys failed in default:scope2:UFoDxZyL-oDFA2kz55A-51-757000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:48,895 | test | INFO | pool-29-thread-22 | [table_view:display:72] Keys failed in default:-9fKRoPO16pojqOZ-42-402000:WgOXH76zBepfiob4h-42-505000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:48,897 | test | INFO | pool-29-thread-22 | [table_view:display:72] Keys failed in default:-9fKRoPO16pojqOZ-42-402000:WgOXH76zBepfiob4h-42-505000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:49,010 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 74.92 2021-05-17 13:46:50,242 | test | INFO | pool-29-thread-29 | [table_view:display:72] Keys failed in default:PCbO-45-975000:wh_K-46-136000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1588 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1458 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1582 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1542 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:50,842 | test | INFO | pool-29-thread-7 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1348 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:52,661 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-603 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-689 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-624 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:53,846 | test | INFO | pool-29-thread-1 | [table_view:display:72] Keys failed in default:_default:0A34Dh1J2tFz-51-749000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1121 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:54,134 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 77.03 2021-05-17 13:46:54,500 | test | INFO | pool-29-thread-3 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-753 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-740 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:54,694 | test | INFO | pool-29-thread-6 | [table_view:display:72] Keys failed in default:scope2:5Oa0RzNN-51-756000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1103 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1164 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:55,046 | test | INFO | pool-29-thread-23 | [table_view:display:72] Keys failed in default:scope2:Y5ag3AT-51-756000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-965 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:58,043 | test | INFO | pool-29-thread-30 | [table_view:display:72] Keys failed in default:_default:Uff-51-748000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1314 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1284 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1275 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:46:59,242 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 78.76 2021-05-17 13:46:59,344 | test | INFO | pool-29-thread-7 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:01,272 | test | INFO | pool-29-thread-2 | [table_view:display:72] Keys failed in default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-988 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:02,099 | test | INFO | pool-29-thread-3 | [table_view:display:72] Keys failed in default:_default:F-51-747000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-911 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-954 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:02,713 | test | INFO | pool-29-thread-1 | [table_view:display:72] Keys failed in default:_default:0A34Dh1J2tFz-51-749000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:04,361 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 80.84 2021-05-17 13:47:06,739 | test | INFO | pool-29-thread-15 | [table_view:display:72] Keys failed in default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:06,740 | test | INFO | pool-29-thread-15 | [table_view:display:72] Keys failed in default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:07,040 | test | INFO | pool-29-thread-6 | [table_view:display:72] Keys failed in default:scope2:5Oa0RzNN-51-756000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1365 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:08,604 | test | INFO | pool-29-thread-18 | [table_view:display:72] Keys failed in default:_default:U6bD__z-51-749000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:09,464 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 82.63 2021-05-17 13:47:12,543 | test | INFO | pool-29-thread-29 | [table_view:display:72] Keys failed in default:PCbO-45-975000:wh_K-46-136000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:14,020 | test | INFO | pool-29-thread-7 | [table_view:display:72] Keys failed in default:PCbO-45-975000:ep4h5iyEg-46-137000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 13:47:14,579 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 84.44 2021-05-17 13:47:19,697 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 86.06 2021-05-17 13:47:24,822 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 88.04 2021-05-17 13:47:29,963 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 90.33 2021-05-17 13:47:35,088 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 92.25 2021-05-17 13:47:40,227 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 94.12 2021-05-17 13:47:45,342 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 96.26 2021-05-17 13:47:50,460 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 98.4 2021-05-17 13:47:56,473 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 13:48:06,625 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 13:48:06,677 | test | INFO | pool-29-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 211.654999971 sec 2021-05-17 13:52:43,657 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 19 docs from 1000 to 1960 2021-05-17 13:52:43,658 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 14 docs from 0 to 1000 2021-05-17 13:52:43,661 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 13:52:43,664 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 13:52:43,664 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 11 docs from 1000 to 1960 2021-05-17 13:52:43,664 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 13:52:43,664 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 13:52:43,665 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 7 docs from 1000 to 1960 2021-05-17 13:52:43,665 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 13:52:43,667 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 16 docs from 0 to 1000 2021-05-17 13:52:43,667 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 5000 to 5800 2021-05-17 13:52:43,667 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 16 docs from 980 to 1440 2021-05-17 13:52:43,667 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 20 docs from 500 to 980 2021-05-17 13:52:43,668 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 13:52:43,670 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 16 docs from 980 to 1440 2021-05-17 13:52:43,671 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 12 docs from 980 to 1440 2021-05-17 13:52:43,671 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 12 docs from 980 to 1440 2021-05-17 13:52:43,674 | test | ERROR | pool-29-thread-17 | [task:call:3809] Failed to load 1 docs from 1000 to 1960 2021-05-17 13:58:48,825 | test | INFO | pool-29-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................*****X | 1255.748 | | 385.238 | .............................****X | 385.238 | | 271.884 | ....................*****.****X | 277.030 | | 175.431 | ............****..*.****......****X | 277.192 | | 96.497 | ......*****..........****........*......*X | 397.933 | | 397.933 | ..............................*X | 397.933 | | 246.747 | ..................*****....***X | 291.310 | | 288.567 | .....................*.****..............****X | 472.767 | | 429.591 | .................................X | 429.591 | | 411.795 | ...............................***X | 411.795 | | 240.158 | ..................****X | 240.158 | | 237.098 | .................*****.......****X | 324.708 | | 199.028 | ..............****X | 199.028 | | 125.761 | .........*****........*.............****X | 396.640 | | 308.407 | .......................****X | 308.407 | | 269.105 | ....................X | 269.105 | | 232.891 | .................****X | 232.891 | | 220.086 | ................X | 220.086 | | 187.097 | .............****X | 187.097 | | 179.972 | .............****X | 179.972 | | 179.383 | .............*..****X | 201.041 | | 183.484 | .............******.****X | 193.436 | | 179.002 | .............******....*****X | 230.488 | | 167.767 | ............****X | 167.767 | | 141.887 | ..........X | 141.887 | | 133.309 | .........*****...........*****X | 265.574 | | 261.401 | ...................***X | 261.401 | | 261.401 | ...................X | 261.401 | | 166.597 | ............****X | 166.597 | | 164.863 | ............X | 164.863 | | 158.363 | ...........*****......*..***X | 254.841 | | 249.997 | ..................X | 249.997 | | 214.361 | ................****X | 214.361 | | 186.774 | .............X | 186.774 | | 181.304 | .............***X | 181.304 | | 168.077 | ............X | 168.077 | | 160.967 | ...........*****......*****X | 230.056 | | 161.899 | ...........****X | 161.899 | | 155.261 | ...........*.**********.....*.***X | 242.666 | | 239.768 | ..................X | 239.768 | | 238.268 | .................*****.****X | 246.906 | | 246.373 | ..................X | 246.373 | | 225.917 | ................*****.*****X | 238.337 | | 165.189 | ............****X | 165.189 | | 164.963 | ............******..........*..***X | 320.898 | | 227.475 | .................X | 227.475 | | 182.379 | .............*****.............****X | 347.265 | | 265.467 | ....................X | 265.467 | | 210.634 | ...............****X | 210.634 | | 190.150 | ..............X | 190.150 | | 175.342 | ............***X | 175.342 | | 168.139 | ............X | 168.139 | | 148.272 | ..........*****.....*****X | 208.331 | | 198.045 | ..............*****..*.....*****............*****.......*.****X | 534.081 | | 263.684 | ...................******.............****X | 423.326 | | 387.237 | .............................X | 387.237 | | 342.285 | ..........................*****.*.*****..****X | 397.591 | | 353.270 | ...........................X | 353.270 | | 281.394 | .....................*********X | 285.717 | | 227.189 | .................****X | 227.189 | | 217.006 | ................X | 217.006 | | 216.015 | ................****X | 216.015 | | 203.622 | ...............*****...........*..*****.....*...***X | 473.247 | | 464.460 | ...................................*.****X | 465.735 | | 295.849 | ......................*****X | 295.849 | | 234.348 | .................*****X | 234.348 | | 234.146 | .................****X | 234.146 | | 218.528 | ................X | 218.528 | | 173.269 | ............*****.........*****X | 278.800 | | 239.983 | ..................****X | 239.983 | | 195.335 | ..............*****X | 195.335 | | 179.362 | .............*****...............******...........X | 511.378 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 13:58:50,118 | test | INFO | pool-29-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *............*......................................*********X | 97.011 | | 1.000 | ****X | 1.000 | | 0.000 | **************.........****.X | 19.798 | | 17.699 | ........****...............****X | 47.685 | | 17.209 | .......***X | 17.209 | | 1.200 | ****X | 1.200 | | 0.400 | ********X | 0.800 | | 0.600 | *****..*****........*****..........................****X | 70.098 | | 55.301 | ...........................****X | 55.301 | | 53.980 | ...........................*****...*****........*.****X | 77.569 | | 69.614 | ...................................*.*****.*****.....X | 84.104 | | 82.499 | ..........................................******.****X | 85.387 | | 83.815 | ..........................................******.........................................................*****X | 191.385 | | 96.517 | .................................................*****............................................*****X | 181.803 | | 82.295 | .........................................*****...*****...*....****X | 99.607 | | 99.496 | ..................................................X | 99.496 | | 94.090 | ................................................*****.......****X | 107.601 | | 86.898 | ............................................****X | 86.898 | | 59.687 | ..............................X | 59.687 | | 54.087 | ...........................*****...............*****...*.............****X | 112.980 | | 98.022 | ..................................................X | 98.022 | | 91.837 | ..............................................****X | 91.837 | | 89.221 | .............................................X | 89.221 | | 78.882 | ........................................****X | 78.882 | | 66.516 | .................................*****..X | 69.601 | | 66.700 | .................................****X | 66.700 | | 59.544 | ..............................******..*****....****X | 70.828 | | 67.350 | ..................................X | 67.350 | | 64.356 | ................................*****.*.****X | 67.853 | | 63.746 | ................................******.......*****....*.*****.****X | 89.525 | | 86.447 | ............................................*..****X | 90.136 | | 89.244 | .............................................X | 89.244 | | 76.259 | ......................................*****.*****.....*.*****..****X | 92.069 | | 91.267 | ..............................................X | 91.267 | | 77.035 | .......................................****X | 77.035 | | 59.781 | ..............................****X | 59.781 | | 55.971 | ............................X | 55.971 | | 54.757 | ...........................****X | 54.757 | | 49.357 | ........................X | 49.357 | | 33.995 | ................****X | 33.995 | | 2.498 | ****X | 2.498 | | 1.000 | ******.........................................****X | 79.663 | | 33.689 | ................X | 33.689 | | 21.291 | ..........****X | 21.291 | | 7.699 | ...X | 7.699 | | 4.198 | .***X | 4.198 | | 1.199 | ******.******.****X | 4.399 | | 0.799 | ***********.......................*.......****X | 57.880 | | 47.483 | .......................****X | 47.483 | | 13.500 | ......X | 13.500 | | 2.800 | ****X | 2.800 | | 1.400 | *****.******...............................****X | 61.392 | | 52.792 | ..........................X | 52.792 | | 40.191 | ...................*****.X | 41.190 | | 41.091 | ....................*****........****X | 55.994 | | 11.895 | .....******...........*****...........******....****X | 61.988 | | 44.893 | ......................X | 44.893 | | 42.291 | .....................*****.........*X | 60.197 | +--------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:58:50,315 | test | INFO | pool-29-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........................*...........................................................................****X | 196.553 | | 78.770 | ......................................X | 78.770 | | 37.805 | ..................*****.............****X | 63.997 | | 62.186 | ..............................***X | 62.186 | | 47.607 | .......................X | 47.607 | | 47.607 | .......................****.......***X | 61.211 | | 38.300 | ..................****X | 38.499 | | 12.300 | .....*****......................*****.....****................................................****X | 161.474 | | 66.519 | ................................****.X | 68.306 | | 62.572 | ..............................****..................................................****X | 161.081 | | 86.500 | ..........................................*****X | 86.500 | | 69.116 | ..................................***X | 69.116 | | 56.116 | ...........................X | 56.116 | | 52.899 | .........................****.......*...***X | 72.201 | | 60.204 | .............................X | 60.204 | | 50.985 | ........................*****..****X | 54.128 | | 53.904 | ..........................*.***X | 56.377 | | 55.689 | ...........................X | 55.689 | | 50.318 | ........................*X | 50.318 | | 50.318 | ........................***..............*****X | 76.997 | | 74.799 | ....................................*****X | 74.799 | | 56.292 | ...........................*****.......................................*.............................****X | 190.935 | | 169.801 | .....................................................................................X | 169.801 | | 93.292 | ..............................................****X | 93.292 | | 60.011 | .............................****X | 60.011 | | 58.701 | ............................******.........*****..*****X | 80.584 | | 70.806 | ..................................*****.*.........***X | 89.392 | | 88.186 | ...........................................*.*****...*****.*****X | 98.578 | | 83.215 | .........................................*****...*.***X | 92.283 | | 92.283 | .............................................*.*****.*....*****.........................................*****X | 184.844 | | 70.499 | ..................................*****.........*.........*****X | 105.202 | | 93.407 | ..............................................*****......*****......................................*.........****X | 197.393 | | 96.318 | ...............................................*****...X | 101.707 | | 98.023 | ................................................X | 98.023 | | 98.023 | ................................................***X | 98.023 | | 83.950 | .........................................******.*...****.X | 94.331 | | 86.737 | ..........................................*****.....**X | 95.433 | | 95.433 | ...............................................**.X | 97.816 | | 94.347 | ..............................................*****.................*.....................***X | 169.283 | | 111.085 | .......................................................X | 111.085 | | 70.448 | ..................................*****..*****X | 74.375 | | 57.353 | ............................******.****X | 59.583 | | 58.562 | ............................X | 58.562 | | 58.354 | ............................****X | 58.354 | | 24.897 | ...........****X | 24.897 | | 4.699 | .X | 4.699 | | 0.799 | **********.*****X | 3.798 | | 1.000 | ******........................*......***X | 60.787 | | 50.886 | ........................X | 50.886 | | 38.559 | ..................*****..*****.............*.....................***X | 108.855 | | 88.763 | ...........................................X | 88.763 | | 56.083 | ...........................*****......*.......................****X | 112.846 | | 83.669 | .........................................*****X | 83.669 | | 75.173 | .....................................****X | 75.173 | | 25.499 | ...........X | 25.499 | | 10.197 | ....****X | 10.197 | | 1.300 | ***********.....................*****..............*...*****........*****X | 92.480 | | 81.687 | ........................................*****....................................*****X | 153.548 | | 112.685 | ........................................................*.*****............****X | 139.556 | | 136.461 | ....................................................................*****X | 136.573 | | 119.067 | ...........................................................X | 119.067 | | 55.487 | ...........................****X | 55.487 | | 0.200 | *X | 0.200 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 13:58:50,358 | test | WARNING | Thread-96501 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:PCbO-45-975000:wh_K-46-136000 update failed keys from task: [u'test_collections-1412', u'test_collections-1588', u'test_collections-1764', u'test_collections-1458', u'test_collections-1418', u'test_collections-1582', u'test_collections-1542'] 2021-05-17 13:58:50,358 | test | WARNING | Thread-96500 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:PCbO-45-975000:ep4h5iyEg-46-137000 update failed keys from task: [u'test_collections-1028'] 2021-05-17 13:58:50,359 | test | WARNING | Thread-96506 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:wVQXxO5-J7DQAketRNzc-51-747000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 13:58:50,364 | test | WARNING | Thread-96517 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:U6bD__z-51-749000 update failed keys from task: [u'test_collections-1183', u'test_collections-1394', u'test_collections-1266', u'test_collections-1244', u'test_collections-1223', u'test_collections-1201'] 2021-05-17 13:58:50,365 | test | WARNING | Thread-96525 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:0A34Dh1J2tFz-51-749000 update failed keys from task: [u'test_collections-1093', u'test_collections-1103', u'test_collections-1109', u'test_collections-1121', u'test_collections-1143', u'test_collections-1000', u'test_collections-1045', u'test_collections-1320', u'test_collections-1059', u'test_collections-1257', u'test_collections-1373', u'test_collections-1099'] 2021-05-17 13:58:50,367 | test | WARNING | Thread-96528 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Uff-51-748000 update failed keys from task: [u'test_collections-1161', u'test_collections-1284', u'test_collections-1320', u'test_collections-1068', u'test_collections-1080', u'test_collections-1314', u'test_collections-1336', u'test_collections-1217', u'test_collections-1074', u'test_collections-1351', u'test_collections-1373', u'test_collections-1110', u'test_collections-1275', u'test_collections-1155', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 13:58:50,368 | test | WARNING | Thread-96534 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:uxE2HFim3uKAL1py3YDB-51-747000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 13:58:50,368 | test | WARNING | Thread-96535 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-51-747000 update failed keys from task: [u'test_collections-1269', u'test_collections-1423', u'test_collections-1401', u'test_collections-1348', u'test_collections-1161', u'test_collections-1360', u'test_collections-1382', u'test_collections-1068', u'test_collections-1387', u'test_collections-1080', u'test_collections-1074', u'test_collections-1031', u'test_collections-1230', u'test_collections-1110', u'test_collections-1155', u'test_collections-1013'] 2021-05-17 13:58:50,369 | test | WARNING | Thread-96537 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:-9fKRoPO16pojqOZ-42-402000:WgOXH76zBepfiob4h-42-505000 create failed keys from task: [u'test_collections-1137'] 2021-05-17 13:58:50,424 | test | WARNING | Thread-96593 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:UFoDxZyL-oDFA2kz55A-51-757000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 13:58:50,453 | test | WARNING | Thread-96580 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:Y5ag3AT-51-756000 delete failed keys from task: [u'test_collections-947', u'test_collections-965'] 2021-05-17 13:58:50,453 | test | WARNING | Thread-96606 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:5Oa0RzNN-51-756000 update failed keys from task: [u'test_collections-1093', u'test_collections-1103', u'test_collections-1204', u'test_collections-1307', u'test_collections-1164', u'test_collections-1186', u'test_collections-1000', u'test_collections-1045', u'test_collections-1365', u'test_collections-1059', u'test_collections-1138', u'test_collections-1297'] 2021-05-17 13:58:50,453 | test | WARNING | Thread-96609 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 delete failed keys from task: [u'test_collections-526', u'test_collections-603', u'test_collections-988', u'test_collections-689', u'test_collections-624', u'test_collections-402', u'test_collections-487', u'test_collections-740', u'test_collections-719', u'test_collections-818', u'test_collections-939', u'test_collections-517', u'test_collections-759', u'test_collections-793', u'test_collections-552', u'test_collections-592'] 2021-05-17 13:58:50,466 | test | WARNING | Thread-96778 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:eSraAKxUCyos-43-995000:AcfTfL%lZ%4PY80BbO2_-44-117000 update failed keys from task: [u'test_collections-1466', u'test_collections-1444', u'test_collections-1565', u'test_collections-1269', u'test_collections-1764', u'test_collections-1325', u'test_collections-1360', u'test_collections-1560', u'test_collections-1244', u'test_collections-1379', u'test_collections-1339'] 2021-05-17 13:58:50,474 | test | WARNING | Thread-96810 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 update failed keys from task: [u'test_collections-1507', u'test_collections-1588', u'test_collections-1764', u'test_collections-1502', u'test_collections-1469', u'test_collections-1547', u'test_collections-1308', u'test_collections-1605', u'test_collections-1582', u'test_collections-1542', u'test_collections-1685', u'test_collections-1412', u'test_collections-1458', u'test_collections-1418', u'test_collections-1715', u'test_collections-1492', u'test_collections-1298', u'test_collections-1497', u'test_collections-1795'] 2021-05-17 13:58:50,782 | test | WARNING | Thread-96609 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:2ohf7eC46-fpZS-45-540000:8%E6b%4BYZgbC-45-665000 create failed keys from task: [u'test_collections-5754'] 2021-05-17 13:58:51,394 | test | WARNING | Thread-96810 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:uM40an3GrCsX9-44-362000:SVwsF%-44-457000 delete failed keys from task: [u'test_collections-529', u'test_collections-265', u'test_collections-541', u'test_collections-220', u'test_collections-482', u'test_collections-535', u'test_collections-335', u'test_collections-512', u'test_collections-476', u'test_collections-575', u'test_collections-433', u'test_collections-254', u'test_collections-570', u'test_collections-370'] 2021-05-17 13:58:51,559 | test | WARNING | Thread-96535 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:F-51-747000 delete failed keys from task: [u'test_collections-649', u'test_collections-606', u'test_collections-823', u'test_collections-866', u'test_collections-566', u'test_collections-740', u'test_collections-563', u'test_collections-719', u'test_collections-939', u'test_collections-759', u'test_collections-911', u'test_collections-954', u'test_collections-610', u'test_collections-632', u'test_collections-753', u'test_collections-677', u'test_collections-650', u'test_collections-892', u'test_collections-793', u'test_collections-552'] 2021-05-17 13:59:05,068 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 14:00:21,993 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 14:00:21,993 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 14:00:42,605 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 14:00:42,605 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 14:01:03,720 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 14:01:03,720 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 14:01:52,463 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1289999485 seconds to complete 2021-05-17 14:01:52,480 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3323.652s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 27 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_27 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_28 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_28', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 28, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 14:02:01,993 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #28 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 14:02:02,696 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #28 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 14:02:03,125 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1011a3f type=client open_count=1 channel=[id: 0x4d888b05, 0.0.0.0/0.0.0.0:50842] timeout=300.0>) 2021-05-17 14:02:23,901 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1011a40 type=client open_count=1 channel=[id: 0xdcb4e377, 0.0.0.0/0.0.0.0:57242] timeout=300.0>) 2021-05-17 14:02:25,918 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1011a41 type=client open_count=1 channel=[id: 0xdadb3cca, 0.0.0.0/0.0.0.0:60027] timeout=300.0>) 2021-05-17 14:02:27,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1011a42 type=client open_count=1 channel=[id: 0x0bf778f2, 0.0.0.0/0.0.0.0:34466] timeout=300.0>) 2021-05-17 14:02:51,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1011a43 type=client open_count=1 channel=[id: 0x1ef163ab, 0.0.0.0/0.0.0.0:42898] timeout=300.0>) 2021-05-17 14:02:53,984 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1011a44 type=client open_count=1 channel=[id: 0xfac6ed7f, 0.0.0.0/0.0.0.0:49669] timeout=300.0>) 2021-05-17 14:02:56,005 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1011a45 type=client open_count=1 channel=[id: 0x03fdf7ce, 0.0.0.0/0.0.0.0:47538] timeout=300.0>) 2021-05-17 14:03:19,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1011a46 type=client open_count=1 channel=[id: 0x63b7258d, 0.0.0.0/0.0.0.0:38488] timeout=300.0>) 2021-05-17 14:03:21,967 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1011a47 type=client open_count=1 channel=[id: 0xb200e525, 0.0.0.0/0.0.0.0:46438] timeout=300.0>) 2021-05-17 14:03:23,984 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1011a48 type=client open_count=1 channel=[id: 0xa4276ff9, 0.0.0.0/0.0.0.0:34292] timeout=300.0>) 2021-05-17 14:03:48,046 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1011a49 type=client open_count=1 channel=[id: 0xcca7c2e8, 0.0.0.0/0.0.0.0:50245] timeout=300.0>) 2021-05-17 14:03:50,065 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1011a4a type=client open_count=1 channel=[id: 0x1b33112d, 0.0.0.0/0.0.0.0:57763] timeout=300.0>) 2021-05-17 14:03:52,084 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1011a4b type=client open_count=1 channel=[id: 0xc09e2414, 0.0.0.0/0.0.0.0:57094] timeout=300.0>) 2021-05-17 14:04:16,283 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1011a4c type=client open_count=1 channel=[id: 0x3612d9c4, 0.0.0.0/0.0.0.0:46052] timeout=300.0>) 2021-05-17 14:04:18,302 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1011a4d type=client open_count=1 channel=[id: 0x46919d47, 0.0.0.0/0.0.0.0:35755] timeout=300.0>) 2021-05-17 14:04:20,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 14:04:37,809 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 14:04:51,552 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 14:04:51,575 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #28 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 14:04:51,615 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #28 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 14:05:05,223 | test | INFO | pool-30-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.94825086249 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 14:05:05,247 | test | INFO | pool-30-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:05:15,326 | test | INFO | pool-30-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 14:05:15,354 | test | INFO | pool-30-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.129999876 sec 2021-05-17 14:05:15,615 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 14:06:03,924 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 14:06:04,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 566032968 | 104302496 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 647004896 | 118324042 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 850879944 | 71908329 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 14:06:04,232 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 14:06:53,667 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 14:06:53,683 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 14:23:52,588 | test | INFO | pool-30-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...................................*......****.................................................*..........****X | 70.039 | | 2.596 | ..X | 2.596 | | 0.000 | *********************************..*****X | 1.501 | | 0.100 | X | 0.100 | | 0.000 | ****.........*.***X | 7.401 | | 5.401 | ......X | 5.401 | | 0.300 | *****....X | 3.101 | | 2.801 | ..***X | 2.801 | | 2.401 | ..X | 2.401 | | 1.500 | .****X | 1.500 | | 0.000 | *****.*.****X | 1.500 | | 0.000 | **********************....*....****X | 5.899 | | 1.500 | .******........*****X | 7.500 | | 6.001 | .......****X | 6.001 | | 1.999 | .X | 1.999 | | 0.000 | *****************....****X | 3.001 | | 1.000 | X | 1.000 | | 0.000 | ***********..*****..******......*****X | 7.501 | | 2.500 | ..X | 2.500 | | 0.000 | *****......**********X | 4.501 | | 0.000 | ****************************....****X | 3.000 | | 2.300 | ..X | 2.300 | | 1.500 | .*****....*****X | 4.500 | | 0.000 | ******.....*.****X | 4.500 | | 0.300 | X | 0.300 | | 0.000 | **********.*.***X | 1.500 | | 1.200 | X | 1.200 | | 0.000 | *****....*****...*...****X | 7.499 | | 0.500 | X | 0.500 | | 0.000 | *****************......****X | 4.501 | | 3.301 | ...X | 3.301 | | 0.000 | ******....*****......****X | 7.500 | | 4.700 | .....X | 4.700 | | 1.500 | .****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | *****..*****X | 1.500 | | 0.000 | ******......****X | 4.500 | | 3.600 | ....**********X | 3.900 | | 2.699 | ..X | 2.699 | | 0.200 | X | 0.200 | | 0.000 | ****.*.****X | 1.500 | | 0.000 | ******...****X | 2.600 | | 2.000 | .X | 2.000 | | 1.900 | .*****.................*****X | 13.500 | | 1.200 | *****.X | 1.800 | | 1.800 | .****X | 1.800 | | 0.100 | *****..X | 1.500 | | 1.400 | *****..*****..*.****X | 4.900 | | 1.700 | .X | 1.700 | | 1.500 | .****X | 1.500 | | 0.000 | ***********.*****X | 1.100 | | 0.400 | *****..****X | 1.499 | | 0.800 | X | 0.800 | | 0.000 | ******....****X | 3.000 | | 1.500 | .*****X | 1.500 | | 0.000 | ************..........****X | 7.600 | | 7.000 | ........X | 7.000 | | 4.201 | ....****X | 4.201 | | 0.200 | ******....*****...*.****X | 6.000 | | 1.999 | .X | 1.999 | | 0.000 | ****......*...................*****X | 18.006 | | 6.000 | .......****X | 6.000 | | 0.400 | X | 0.400 | | 0.000 | *************************X | 0.000 | +-------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 14:23:53,664 | test | INFO | pool-30-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...*.****....................................................*............................................****X | 96.346 | | 5.990 | .....X | 5.990 | | 0.000 | ********************************....*..****X | 6.003 | | 0.000 | ******.*****...******X | 4.601 | | 4.401 | ...***X | 4.401 | | 4.001 | ...X | 4.001 | | 3.000 | ..****X | 3.000 | | 1.500 | *****..........*....****X | 15.001 | | 0.999 | X | 0.999 | | 0.000 | ****************....******.*****X | 4.999 | | 4.501 | ...****X | 4.501 | | 4.099 | ...******.******..*.****X | 8.501 | | 0.000 | ***********..*.****X | 3.001 | | 0.000 | *****************.*****.*..****X | 4.501 | | 1.500 | X | 1.500 | | 0.000 | ****...*.****X | 4.500 | | 1.700 | X | 1.700 | | 1.500 | *****..****X | 3.400 | | 2.500 | .X | 2.500 | | 1.100 | ****X | 1.100 | | 0.000 | ***********.....*.****X | 6.000 | | 3.200 | ..X | 3.200 | | 3.000 | ..****X | 3.000 | | 1.500 | *****X | 1.500 | | 0.000 | ***********..*.****X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | **********.*****X | 1.500 | | 0.000 | *****........*.*****...****X | 11.999 | | 8.799 | ........X | 8.799 | | 0.000 | *****************.*****X | 1.500 | | 0.000 | ******.*********X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ***********************....****X | 4.500 | | 3.600 | ..X | 3.600 | | 0.000 | *****...*****X | 3.000 | | 0.000 | ******.****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | ***********.*****....*.....****X | 10.500 | | 4.101 | ...******.X | 4.899 | | 4.899 | ....***X | 4.899 | | 0.000 | *****************.......****X | 7.399 | | 1.800 | X | 1.800 | | 1.500 | ****X | 1.500 | | 0.000 | ******..*.***X | 3.000 | | 0.200 | X | 0.200 | | 0.000 | ****************.******.........****X | 10.500 | | 5.600 | ....X | 5.600 | | 0.000 | ***********..*****X | 2.300 | | 1.800 | X | 1.800 | | 1.500 | ***X | 1.500 | | 1.300 | X | 1.300 | | 1.300 | ****X | 1.300 | | 1.000 | X | 1.000 | | 0.200 | *****.******........****X | 9.000 | | 0.600 | X | 0.600 | | 0.000 | *****.*****X | 1.501 | | 0.000 | ********************************X | 0.000 | +-------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 14:23:54,529 | test | INFO | pool-30-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******........................................................................................*****..........*..****X | 795.119 | | 527.170 | .................................................................X | 527.170 | | 453.193 | .......................................................*****.................*.....****X | 624.736 | | 595.870 | .........................................................................*****..*......****X | 656.874 | | 588.520 | .........................................................................X | 588.520 | | 566.580 | ......................................................................***X | 566.580 | | 549.269 | ....................................................................X | 549.269 | | 527.723 | .................................................................*****...****X | 552.886 | | 523.768 | ................................................................X | 523.768 | | 506.086 | ..............................................................***X | 506.086 | | 491.368 | ............................................................X | 491.368 | | 465.185 | .........................................................*****.......*.*****.........****X | 600.358 | | 487.179 | ............................................................X | 487.179 | | 463.309 | .........................................................****X | 463.309 | | 408.261 | ..................................................*****...*..........****X | 511.288 | | 466.278 | .........................................................*..*****..................*****X | 623.250 | | 517.884 | ................................................................****X | 517.884 | | 508.775 | ..............................................................*....****X | 535.093 | | 504.542 | ..............................................................*****X | 504.542 | | 450.860 | .......................................................****X | 450.860 | | 397.138 | ................................................X | 397.138 | | 379.607 | ..............................................*****......*..*****.*****.....*..*****...*****...*..****X | 570.107 | | 492.628 | ............................................................X | 492.628 | | 469.228 | ..........................................................****........*.****X | 548.236 | | 518.066 | ................................................................*****X | 518.066 | | 424.452 | ....................................................*****..................*****X | 572.393 | | 478.778 | ...........................................................X | 478.778 | | 471.691 | ..........................................................***X | 471.691 | | 468.581 | .........................................................*......****X | 510.758 | | 455.789 | ........................................................X | 455.789 | | 454.661 | ........................................................****X | 454.661 | | 437.478 | ......................................................******..X | 457.962 | | 455.591 | ........................................................****..*.....****X | 509.102 | | 462.038 | .........................................................*.*****X | 474.993 | | 469.195 | ..........................................................****X | 469.195 | | 448.067 | .......................................................****X | 448.067 | | 435.789 | .....................................................******..****X | 451.195 | | 446.295 | .......................................................X | 446.295 | | 440.462 | ......................................................****X | 440.462 | | 424.511 | ....................................................X | 424.511 | | 416.406 | ...................................................*****......*.****.....X | 512.940 | | 501.992 | ..............................................................******.****X | 508.964 | | 471.804 | ..........................................................X | 471.804 | | 460.004 | ........................................................****X | 460.004 | | 369.955 | .............................................*****.................*****X | 508.242 | | 497.572 | .............................................................*.***X | 508.361 | | 492.017 | ............................................................X | 492.017 | | 479.823 | ...........................................................*****.......*.*****.....*****X | 582.876 | | 464.106 | .........................................................****X | 464.106 | | 423.935 | ....................................................X | 423.935 | | 420.909 | ...................................................*****.............****X | 519.057 | | 484.961 | ...........................................................X | 484.961 | | 464.320 | .........................................................*****.X | 470.689 | | 463.871 | .........................................................*****.......*..****X | 536.858 | | 514.012 | ...............................................................******.*..***X | 534.326 | | 511.749 | ...............................................................X | 511.749 | | 491.836 | ............................................................****X | 491.836 | | 469.585 | ..........................................................*.**********X | 481.480 | | 427.588 | ....................................................*****.....*.*****.****X | 482.710 | | 467.934 | .........................................................X | 467.934 | | 417.190 | ...................................................X | 417.190 | | 409.555 | ..................................................****.*.......*****....****X | 504.497 | | 464.670 | .........................................................X | 464.670 | | 430.954 | .....................................................*****......*****X | 480.908 | | 422.997 | ....................................................*****..X | 438.743 | | 433.352 | .....................................................****X | 433.352 | | 393.717 | ................................................X | 393.717 | | 392.609 | ................................................*****......*****....*****X | 472.246 | | 421.336 | ...................................................******................****X | 545.604 | | 514.253 | ...............................................................*****.X | 521.596 | | 485.225 | ............................................................*****X | 492.490 | | 418.904 | ...................................................*****.........****X | 490.117 | | 443.566 | ......................................................X | 443.566 | | 436.165 | .....................................................*****....*****X | 466.444 | | 412.619 | ..................................................******......*..****...X | 494.222 | | 482.911 | ...........................................................****X | 482.911 | | 471.420 | ..........................................................******.X | 481.669 | | 471.195 | ..........................................................****X | 471.195 | | 415.674 | ...................................................*****.X | 426.876 | | 414.977 | ...................................................*****..*...***********.........*.***X | 535.259 | | 516.635 | ...............................................................X | 516.635 | | 463.369 | .........................................................X | 463.369 | | 463.049 | .........................................................****.........................*.....****X | 700.287 | | 613.928 | ............................................................................X | 613.928 | | 611.242 | ...........................................................................***X | 611.242 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:23:54,802 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 62.995 | 6437847040 | 4322250752 | 2413580288 / 7511588864 | 101487 / 202307 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 70.5476484117 | 6437847040 | 4228317184 | 2480726016 / 7511588864 | 99661 / 203860 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 68.7261454358 | 6437847040 | 4263133184 | 2430595072 / 7511588864 | 100142 / 199293 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 74.7408246942 | 6437847040 | 3842674688 | 3048771584 / 9530753024 | 101291 / 199378 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 74.7637381869 | 6437847040 | 4257210368 | 2453647360 / 7511588864 | 100790 / 201899 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 14:24:07,632 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 14:25:34,914 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 165523592 | 249035565 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162362656 | 268620162 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 443738552 | 454871793 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 14:25:34,914 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #28 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 14:25:45,065 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 14:25:45,065 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 14:25:45,066 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 14:26:27,112 | test | INFO | pool-30-thread-3 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 41.8499999046 seconds to complete 2021-05-17 14:26:27,118 | test | INFO | pool-30-thread-3 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 14:28:32,197 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0730001926422 seconds 2021-05-17 14:28:32,197 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 14:28:32,197 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 14:28:40,328 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 14:41:06,154 | test | INFO | pool-30-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.******.............******..................................******................****X | 172.050 | | 164.581 | ............................................................X | 164.581 | | 144.266 | ....................................................****X | 144.266 | | 131.467 | ...............................................******.................*****X | 174.739 | | 144.080 | ....................................................******.......******..............****X | 199.746 | | 197.044 | ........................................................................X | 197.044 | | 188.149 | .....................................................................****X | 188.149 | | 180.165 | ..................................................................X | 180.165 | | 169.847 | ..............................................................*****...X | 177.479 | | 169.268 | ..............................................................****X | 169.268 | | 164.873 | ............................................................*...........*****.........******...................*****X | 268.449 | | 15.697 | ....****X | 15.697 | | 1.800 | X | 1.800 | | 1.299 | ****X | 1.299 | | 0.500 | ******.****X | 3.599 | | 2.000 | X | 2.000 | | 0.400 | ****X | 0.400 | | 0.200 | *****X | 0.200 | | 0.100 | X | 0.100 | | 0.000 | ****************X | 1.500 | | 0.400 | X | 0.400 | | 0.200 | **********...................*.......*****...**.****X | 80.906 | | 76.604 | ...........................*****.*.****X | 81.453 | | 81.071 | .............................X | 81.071 | | 71.472 | .........................*****.X | 73.265 | | 65.859 | .......................****X | 65.859 | | 63.258 | ......................X | 63.258 | | 61.506 | .....................****X | 61.506 | | 16.246 | .....X | 16.246 | | 2.402 | *****X | 2.402 | | 0.500 | **********X | 2.102 | | 1.601 | ******.*****X | 3.300 | | 1.700 | *****X | 1.700 | | 1.001 | ***********X | 1.399 | | 1.199 | ******..*****X | 6.100 | | 0.200 | **********X | 1.201 | | 0.901 | X | 0.901 | | 0.600 | **********X | 1.599 | | 0.700 | **********X | 2.399 | | 1.898 | X | 1.898 | | 1.298 | X | 1.298 | | 0.799 | ***X | 0.799 | | 0.499 | *X | 0.499 | | 0.300 | ***X | 0.300 | | 0.200 | *X | 0.200 | | 0.200 | ***************X | 2.399 | | 0.898 | X | 0.898 | | 0.000 | *********X | 1.500 | | 1.200 | *X | 1.200 | | 1.100 | **********X | 1.500 | | 0.899 | X | 0.899 | | 0.600 | ****X | 0.600 | | 0.200 | X | 0.200 | | 0.000 | **********X | 1.800 | | 1.499 | ***********X | 2.299 | | 0.600 | **********************X | 1.799 | | 1.699 | X | 1.699 | | 0.900 | ****X | 0.900 | | 0.700 | X | 0.700 | | 0.500 | **********X | 2.200 | | 2.100 | ****X | 2.499 | | 1.399 | *X | 1.399 | | 0.800 | *********X | 1.099 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:41:06,499 | test | INFO | pool-30-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.******...........*....*****.............*................................*****..............*****X | 187.347 | | 174.858 | .....................................................................****X | 174.858 | | 168.466 | ..................................................................X | 168.466 | | 147.761 | ..........................................................*****.......*.......................****X | 223.123 | | 216.523 | ......................................................................................X | 216.523 | | 208.174 | ..................................................................................****X | 208.174 | | 205.769 | .................................................................................X | 205.769 | | 171.347 | ...................................................................*****.......................******...X | 234.843 | | 234.737 | .............................................................................................****X | 234.737 | | 173.248 | ....................................................................X | 173.248 | | 155.254 | .............................................................*****.......................*****..*....****X | 227.763 | | 174.263 | .....................................................................X | 174.263 | | 148.463 | ..........................................................*****.................*....................*****....*****X | 248.484 | | 134.190 | .....................................................X | 134.190 | | 90.079 | ...................................****X | 90.079 | | 63.994 | ........................******......****X | 78.380 | | 74.384 | ............................X | 74.384 | | 72.400 | ............................*****.*.****X | 78.485 | | 72.795 | ............................X | 72.795 | | 66.595 | .........................*****...X | 72.489 | | 71.191 | ...........................****X | 71.191 | | 25.096 | .........*****X | 25.096 | | 0.200 | ***********.*..*****X | 7.606 | | 1.999 | *****X | 1.999 | | 0.400 | ***********X | 0.759 | | 0.200 | ************.******.*****X | 6.808 | | 0.802 | *****X | 0.802 | | 0.401 | ***********.*.*****X | 5.901 | | 4.804 | ****X | 4.804 | | 2.604 | X | 2.604 | | 0.200 | **********X | 1.100 | | 1.000 | X | 1.000 | | 0.699 | *****...*.****X | 11.802 | | 0.300 | **********X | 0.701 | | 0.501 | X | 0.501 | | 0.200 | *****************...*****X | 7.698 | | 3.094 | X | 3.094 | | 0.799 | ***********X | 1.001 | | 0.700 | ****X | 0.700 | | 0.700 | ******..*.****X | 8.696 | | 3.194 | X | 3.194 | | 0.100 | ****X | 0.200 | | 0.100 | ***********.**.***X | 6.299 | | 3.097 | X | 3.097 | | 1.400 | *****X | 1.400 | | 0.600 | ************..****X | 7.099 | | 3.396 | X | 3.396 | | 1.499 | *****X | 1.499 | | 0.200 | **********X | 2.200 | | 2.200 | *.****X | 2.699 | | 2.398 | X | 2.398 | | 1.899 | X | 1.899 | | 1.199 | *****.****X | 3.298 | | 1.398 | X | 1.398 | | 0.000 | ******.***X | 3.198 | | 2.999 | *X | 2.999 | | 2.399 | ***X | 2.399 | | 2.199 | *****X | 2.399 | | 1.999 | X | 1.999 | | 1.299 | ****X | 1.299 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:41:06,619 | test | INFO | pool-30-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.*.*****............*..........*****......................*......................................................****X | 1779.670 | | 431.497 | .......................*****..........*............................****X | 1108.622 | | 546.067 | .............................X | 546.067 | | 350.104 | ..................****X | 350.104 | | 286.441 | ...............X | 286.441 | | 273.903 | ..............****X | 273.903 | | 261.903 | .............X | 261.903 | | 240.168 | ............****X | 240.168 | | 212.432 | ..........******..******..*****.******...........****X | 492.001 | | 391.499 | ....................X | 391.499 | | 333.448 | .................*****..*.****X | 384.495 | | 308.339 | ................*****X | 308.339 | | 301.782 | ...............*****..........*....****X | 536.875 | | 370.781 | ...................X | 370.781 | | 319.875 | ................*****......******......****X | 521.712 | | 510.096 | ...........................X | 510.096 | | 457.407 | ........................****X | 457.407 | | 320.591 | .................X | 320.591 | | 284.080 | ..............******..*****.*......*****........*.*****.....*..***X | 714.381 | | 714.381 | .......................................X | 714.381 | | 699.950 | ......................................X | 699.950 | | 681.364 | .....................................****X | 681.364 | | 530.338 | ............................X | 530.338 | | 449.901 | ........................****X | 449.901 | | 347.331 | ..................X | 347.331 | | 316.716 | ................*****........*...X | 502.523 | | 502.523 | ...........................**X | 502.523 | | 502.523 | ...........................X | 502.523 | | 490.715 | ..........................X | 490.715 | | 483.714 | ..........................*****.******..*****X | 544.169 | | 528.542 | ............................******....***********.X | 608.764 | | 586.790 | ...............................*X | 586.790 | | 586.790 | ...............................***......*.****X | 708.409 | | 543.380 | .............................******..........*****X | 719.065 | | 422.540 | ......................******....*****.*.*****........*****X | 667.501 | | 593.964 | ................................******X | 599.249 | | 589.651 | ................................****X | 589.651 | | 542.399 | .............................X | 542.399 | | 500.377 | ...........................****X | 500.377 | | 489.644 | ..........................X | 489.644 | | 472.204 | .........................****X | 472.204 | | 370.357 | ...................X | 370.357 | | 341.353 | ..................****...*......*...***X | 563.126 | | 520.450 | ............................X | 520.450 | | 501.926 | ...........................X | 501.926 | | 472.355 | .........................****.*.*.***X | 525.716 | | 512.225 | ...........................X | 512.225 | | 503.722 | ...........................*****.**.***X | 542.467 | | 451.475 | ........................X | 451.475 | | 419.286 | ......................*****.*****..*......****X | 574.698 | | 420.500 | ......................X | 420.500 | | 375.914 | ....................*****..*...*..***X | 506.886 | | 485.165 | ..........................*X | 485.165 | | 470.490 | .........................***X | 470.490 | | 465.211 | .........................*****X | 469.416 | | 430.509 | .......................X | 430.509 | | 425.229 | ......................*****....******...*..**********X | 578.164 | | 496.885 | ..........................*****.*..****X | 551.314 | | 479.452 | .........................X | 479.452 | | 411.624 | ......................*****..*....*****..******.*......*****X | 684.662 | | 682.864 | .....................................*****..*.****X | 733.383 | +---------+--------------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 14:41:06,882 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 14:41:06,944 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 14:41:17,362 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 14:41:17,638 | test | INFO | pool-30-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.389993500108 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 37.0029000967 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 30.8611523141 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 67.655 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 47.4225429576 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 14:41:17,719 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:41:22,815 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:41:25,114 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 14:41:27,908 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:41:33,016 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:41:38,219 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 14:41:43,319 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.18 2021-05-17 14:41:48,431 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 12.44 2021-05-17 14:41:53,545 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 18.97 2021-05-17 14:41:55,421 | test | INFO | pool-30-thread-13 | [table_view:display:72] Keys failed in bucket1:2GXKs_zVV_P0YoE-22-813000:KPOljXF31in-22-961000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:55,423 | test | INFO | pool-30-thread-21 | [table_view:display:72] Keys failed in bucket1:pv80sO-22-468000:1%FSJ5Z1N_8lccBZRHG-22-599000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:55,423 | test | INFO | pool-30-thread-13 | [table_view:display:72] Keys failed in bucket1:2GXKs_zVV_P0YoE-22-813000:KPOljXF31in-22-961000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:55,421 | test | INFO | pool-30-thread-18 | [table_view:display:72] Keys failed in bucket1:wxdH8TjwVXb-23-106000:c1wUCbhm-23-199000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:55,423 | test | INFO | pool-30-thread-21 | [table_view:display:72] Keys failed in bucket1:pv80sO-22-468000:1%FSJ5Z1N_8lccBZRHG-22-599000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:55,424 | test | INFO | pool-30-thread-18 | [table_view:display:72] Keys failed in bucket1:wxdH8TjwVXb-23-106000:c1wUCbhm-23-199000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:41:58,635 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 20.63 2021-05-17 14:42:03,730 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 23.05 2021-05-17 14:42:08,848 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 26.49 2021-05-17 14:42:13,921 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 29.94 2021-05-17 14:42:19,009 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 37.21 2021-05-17 14:42:21,430 | test | INFO | pool-30-thread-11 | [table_view:display:72] Keys failed in bucket1:wY3QB1lE9C-37-311000:vp5mG_MiID1O1XU8_34h-37-470000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1074 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:21,842 | test | INFO | pool-30-thread-28 | [table_view:display:72] Keys failed in bucket1:q-15-684000:H6X-24-825000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:21,845 | test | INFO | pool-30-thread-28 | [table_view:display:72] Keys failed in bucket1:q-15-684000:H6X-24-825000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-202 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-392 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:24,128 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 49.41 2021-05-17 14:42:29,250 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 53.62 2021-05-17 14:42:34,342 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 55.15 2021-05-17 14:42:39,414 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 57.15 2021-05-17 14:42:44,512 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 58.39 2021-05-17 14:42:48,802 | test | INFO | pool-30-thread-17 | [table_view:display:72] Keys failed in bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1848 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1807 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1918 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1929 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1842 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1833 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1912 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1802 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1882 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:49,638 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 67.98 2021-05-17 14:42:50,589 | test | INFO | pool-30-thread-6 | [table_view:display:72] Keys failed in bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5630 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5760 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:50,591 | test | INFO | pool-30-thread-6 | [table_view:display:72] Keys failed in bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5630 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5760 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:51,936 | test | INFO | pool-30-thread-24 | [table_view:display:72] Keys failed in bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-902 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-947 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-801 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-835 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-844 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-991 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:54,717 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 73.59 2021-05-17 14:42:56,835 | test | INFO | pool-30-thread-10 | [table_view:display:72] Keys failed in default:_default:lOiYeU73--15-686000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-936 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:58,907 | test | INFO | pool-30-thread-9 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:5B-15-698000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1258 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:42:59,825 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 83.53 2021-05-17 14:43:02,174 | test | INFO | pool-30-thread-26 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:WVZLtpYQtDT-15-700000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1258 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1263 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:04,002 | test | INFO | pool-30-thread-7 | [table_view:display:72] Keys failed in default:ZBc3ougCBv11YHy0EY-39-849000:Sp-41dhqDRf-39-977000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-402 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:04,937 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 87.25 2021-05-17 14:43:05,102 | test | INFO | pool-30-thread-16 | [table_view:display:72] Keys failed in default:_default:DK5cIbCBLiu2RnJ%0-15-689000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:05,104 | test | INFO | pool-30-thread-29 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:4uwrqog-15-699000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:05,795 | test | INFO | pool-30-thread-19 | [table_view:display:72] Keys failed in default:_default:m%XWgVMc2hPIX-15-688000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1263 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:06,953 | test | INFO | pool-30-thread-9 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:5B-15-698000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:10,055 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: running, progress: 90.0 2021-05-17 14:43:10,815 | test | INFO | pool-30-thread-25 | [table_view:display:72] Keys failed in default:ZBc3ougCBv11YHy0EY-39-849000:Sp-41dhqDRf-39-977000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1622 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1659 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:11,631 | test | INFO | pool-30-thread-26 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:WVZLtpYQtDT-15-700000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1429 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:13,336 | test | INFO | pool-30-thread-19 | [table_view:display:72] Keys failed in default:_default:m%XWgVMc2hPIX-15-688000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1423 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1417 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:13,575 | test | INFO | pool-30-thread-7 | [table_view:display:72] Keys failed in default:ZBc3ougCBv11YHy0EY-39-849000:Sp-41dhqDRf-39-977000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-759 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-637 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-756 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-672 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:13,595 | test | INFO | pool-30-thread-17 | [table_view:display:72] Keys failed in default:sz3oAGnsT0UK95z-15-698000:WXXSVZsOcnpq9EF-15-700000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1132 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:14,400 | test | INFO | pool-30-thread-16 | [table_view:display:72] Keys failed in default:ZBc3ougCBv11YHy0EY-39-849000:-NfIU-39-976000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:19,276 | test | INFO | pool-30-thread-6 | [table_view:display:72] Keys failed in default:ZBc3ougCBv11YHy0EY-39-849000:-NfIU-39-976000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-99 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-45 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-193 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 14:43:20,163 | test | INFO | pool-30-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 14:43:20,203 | test | INFO | pool-30-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 122.565000057 sec 2021-05-17 14:44:29,861 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 14:44:29,862 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 14 docs from 1000 to 1960 2021-05-17 14:44:29,862 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 11 docs from 0 to 1000 2021-05-17 14:44:29,864 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 14:44:29,864 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 14:45:32,821 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 14:45:32,822 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 2 docs from 5000 to 5800 2021-05-17 14:45:32,822 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 14:45:32,822 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 14:45:32,822 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 6 docs from 0 to 1000 2021-05-17 14:45:32,823 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 14:45:32,825 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 14:45:32,825 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 14:45:32,826 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 3 docs from 980 to 1440 2021-05-17 14:45:32,826 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 10 docs from 980 to 1440 2021-05-17 14:45:32,826 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 8 docs from 980 to 1440 2021-05-17 14:46:22,681 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 7 docs from 980 to 1440 2021-05-17 14:46:22,684 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 14:46:22,684 | test | ERROR | pool-30-thread-8 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 14:56:13,891 | test | INFO | pool-30-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****.........****................****......................****X | 194.508 | | 96.998 | ......................****X | 96.998 | | 29.097 | ......****.......*...****.*****........****................*.....**X | 193.662 | | 139.294 | .................................*****....X | 156.575 | | 150.030 | ...................................****.............*....................................****X | 347.799 | | 321.586 | ..............................................................................*****...........*.........****X | 405.824 | | 396.413 | ................................................................................................****X | 396.413 | | 316.121 | ............................................................................X | 316.121 | | 307.897 | ..........................................................................*****.........................X | 406.805 | | 392.819 | ...............................................................................................****X | 392.819 | | 295.640 | .......................................................................****X | 295.640 | | 260.132 | ..............................................................*..X | 268.216 | | 268.216 | ................................................................***X | 268.216 | | 53.296 | ............****X | 53.296 | | 48.592 | ..........X | 48.592 | | 31.907 | ......X | 31.907 | | 27.506 | .....***X | 27.506 | | 23.206 | ....X | 23.206 | | 3.800 | *****..........****X | 42.002 | | 38.598 | ........****X | 38.801 | | 31.101 | ......X | 31.101 | | 0.000 | *****.........*****..*.****X | 52.301 | | 11.300 | .*****X | 11.300 | | 0.700 | X | 0.700 | | 0.000 | *********X | 0.900 | | 0.700 | ****X | 0.700 | | 0.400 | **********X | 1.600 | | 0.000 | *****************..........*****X | 43.607 | | 40.696 | .........X | 40.696 | | 36.595 | .......****X | 36.595 | | 6.099 | X | 6.099 | | 0.200 | *********X | 2.000 | | 1.000 | ****X | 1.000 | | 0.300 | X | 0.300 | | 0.200 | **********X | 1.800 | | 0.400 | ****X | 0.400 | | 0.400 | *****..............*.........****X | 96.986 | | 0.800 | **********X | 1.800 | | 1.600 | *****X | 1.600 | | 0.500 | X | 0.500 | | 0.000 | *********************X | 1.000 | | 0.800 | ****X | 0.800 | | 0.400 | ****************X | 2.100 | | 0.100 | ****X | 0.100 | | 0.000 | **********X | 2.000 | | 1.700 | X | 1.700 | | 0.000 | *********X | 1.200 | | 1.200 | *****X | 1.200 | | 0.200 | ****************X | 0.600 | | 0.600 | ***********X | 1.800 | | 0.200 | ******.........*****..........*....****X | 94.002 | | 63.801 | ..............*****X | 63.801 | | 0.400 | ***********X | 1.000 | | 0.400 | ****X | 0.400 | | 0.400 | **********X | 1.001 | | 0.800 | ***********X | 0.800 | | 0.800 | ****X | 0.800 | | 0.600 | X | 0.600 | | 0.400 | **********X | 1.400 | | 1.000 | *****X | 1.000 | | 0.200 | *****************.........******.***X | 43.153 | +---------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:56:15,079 | test | INFO | pool-30-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.........................*...........................................................................***X | 202.030 | | 90.888 | ...........................................X | 90.888 | | 50.399 | .......................***....*.........****.......*****..............................................***X | 182.201 | | 96.206 | ..............................................X | 96.206 | | 66.873 | ................................***X | 66.873 | | 26.897 | ............X | 26.897 | | 14.803 | ......****...*......................***X | 66.071 | | 54.213 | .........................****...............****....*..*****...............*.....***X | 136.200 | | 122.898 | ...........................................................X | 122.898 | | 115.196 | ........................................................***X | 115.196 | | 98.188 | ...............................................X | 98.188 | | 93.505 | .............................................*****........*****...*.......*****....*****X | 138.339 | | 131.475 | ................................................................****X | 131.475 | | 130.412 | ...............................................................*****.X | 131.980 | | 122.790 | ...........................................................*****..........*****.................*............****X | 200.184 | | 148.907 | ........................................................................****X | 148.907 | | 147.700 | ........................................................................******.........****X | 167.507 | | 145.306 | ......................................................................****X | 145.306 | | 122.506 | ...........................................................X | 122.506 | | 106.705 | ...................................................*****............*****X | 129.382 | | 126.375 | .............................................................****X | 126.375 | | 121.822 | ...........................................................X | 121.822 | | 119.632 | ..........................................................****X | 119.632 | | 103.502 | ..................................................X | 103.502 | | 102.491 | .................................................***X | 102.491 | | 98.195 | ...............................................X | 98.195 | | 74.508 | ...................................*****.........................................................*X | 189.789 | | 189.789 | ............................................................................................**X | 189.789 | | 143.583 | ......................................................................X | 143.583 | | 81.705 | .......................................******....****X | 89.990 | | 77.795 | .....................................X | 77.795 | | 77.795 | .....................................***X | 77.795 | | 67.006 | ................................X | 67.006 | | 62.110 | .............................*****......................................*.........****X | 156.878 | | 99.112 | ................................................X | 99.112 | | 87.012 | ..........................................***X | 87.012 | | 85.515 | .........................................X | 85.515 | | 80.598 | ......................................X | 80.598 | | 74.502 | ...................................****.*...****X | 80.903 | | 0.800 | ****X | 0.800 | | 0.800 | ************.................................*.........***X | 86.300 | | 72.299 | ..................................X | 72.299 | | 18.497 | ........****X | 18.497 | | 1.200 | ******.****X | 2.500 | | 1.500 | ******...........................****X | 56.090 | | 46.394 | .....................X | 46.394 | | 41.298 | ...................****X | 41.298 | | 7.999 | ..X | 7.999 | | 1.000 | ****X | 1.000 | | 0.400 | ******.*****.............*....*****....*.......****X | 60.592 | | 3.800 | X | 3.800 | | 3.800 | ***X | 3.800 | | 1.800 | ****X | 1.800 | | 1.400 | X | 1.400 | | 0.800 | *****.*****...........*..............*****...............*.......***X | 97.689 | | 97.689 | ...............................................X | 97.689 | | 31.797 | ..............****X | 31.797 | | 7.200 | ..X | 7.200 | | 2.201 | ***********...........................................*****.*X | 92.197 | | 91.395 | ............................................***X | 91.395 | | 21.297 | .........X | 21.297 | | 3.800 | *****..............................*........****X | 79.803 | | 32.598 | ...............X | 32.598 | | 1.600 | ****X | 1.600 | | 1.400 | ****X | 1.400 | | 1.200 | X | 1.200 | | 1.100 | ***********.*****.............*......................****X | 72.887 | | 51.295 | ........................X | 51.295 | | 9.799 | ...***X | 9.799 | | 5.799 | .X | 5.799 | | 3.702 | ****X | 3.702 | | 2.900 | X | 2.900 | | 1.800 | *****.X | 2.400 | | 2.100 | **********X | 2.300 | | 1.200 | ******.****X | 2.813 | | 1.799 | *****X | 1.799 | | 1.600 | ******.****X | 2.400 | | 2.100 | *****X | 2.800 | | 1.900 | ******.X | 2.300 | | 2.100 | ***X | 2.100 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:56:16,085 | test | INFO | pool-30-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 1.200 | X | 1.200 | | 0.000 | ****.........................................***X | 330.952 | | 41.996 | ....****.....*.............***X | 184.112 | | 171.111 | ....................X | 171.111 | | 164.691 | ...................****......*.**X | 219.807 | | 162.472 | ...................X | 162.472 | | 148.042 | .................**X | 148.042 | | 148.042 | .................X | 148.042 | | 55.897 | ......X | 55.897 | | 24.511 | ..*****....****.....................*X | 222.485 | | 222.485 | ...........................**..............****...................*.......*****.........................****X | 741.902 | | 440.963 | ......................................................X | 440.963 | | 367.977 | .............................................****X | 367.977 | | 305.964 | .....................................X | 305.964 | | 300.816 | ....................................***X | 300.816 | | 299.215 | ....................................X | 299.215 | | 282.098 | ..................................*****.............................................................**X | 767.358 | | 767.358 | ...............................................................................................**X | 767.358 | | 469.222 | ..........................................................****X | 469.222 | | 430.704 | .....................................................X | 430.704 | | 406.721 | ..................................................****X | 406.721 | | 394.658 | ................................................*..****.........*****X | 482.959 | | 358.353 | ............................................*****.................................*......................****X | 793.941 | | 466.925 | .........................................................****X | 466.925 | | 321.292 | .......................................X | 321.292 | | 318.200 | .......................................***X | 318.200 | | 300.517 | ....................................X | 300.517 | | 259.005 | ...............................*****....*..............****X | 397.617 | | 248.309 | ..............................*****.......*.......*...***X | 387.348 | | 294.516 | ....................................X | 294.516 | | 272.246 | .................................*****.........................****X | 472.425 | | 437.017 | ......................................................X | 437.017 | | 376.570 | ..............................................*****..........*****..****X | 474.772 | | 443.860 | ......................................................X | 443.860 | | 343.376 | ..........................................X | 343.376 | | 315.620 | ......................................****.........*.****X | 391.557 | | 338.476 | .........................................*****...*....*****X | 396.554 | | 378.048 | ..............................................****X | 378.048 | | 324.541 | .......................................*X | 324.541 | | 324.541 | .......................................***.*..****X | 341.907 | | 339.907 | .........................................X | 339.907 | | 327.815 | ........................................*****...................****X | 478.091 | | 475.881 | ..........................................................*...**X | 494.809 | | 494.809 | .............................................................**....****X | 529.653 | | 246.000 | .............................******..............................*.........****.......X | 608.138 | | 589.830 | .........................................................................*****.....*****X | 632.901 | | 507.633 | ..............................................................****X | 507.633 | | 328.881 | ........................................X | 328.881 | | 288.049 | ...................................X | 288.049 | | 288.049 | ...................................****...........X | 377.175 | | 377.175 | ..............................................**X | 377.175 | | 377.175 | ..............................................**......X | 423.313 | | 422.567 | ....................................................*****..*****...X | 466.091 | | 455.500 | ........................................................****X | 455.500 | | 430.301 | .....................................................X | 430.301 | | 400.184 | .................................................***X | 400.184 | | 349.452 | ...........................................*****......*........****X | 461.963 | | 449.710 | .......................................................X | 449.710 | | 449.191 | .......................................................****....X | 476.864 | | 448.047 | .......................................................****X | 448.047 | | 432.973 | .....................................................X | 432.973 | | 404.488 | .................................................****X | 404.488 | | 364.993 | ............................................X | 364.993 | | 310.732 | ......................................****X | 310.732 | | 236.179 | ............................*****.............*....*****.........******.........****X | 511.525 | | 343.784 | ..........................................X | 343.784 | | 295.289 | ....................................****X | 295.289 | | 225.186 | ...........................******..........................*****X | 433.501 | | 336.014 | .........................................*****...*****X | 364.658 | | 267.576 | ................................X | 267.576 | | 267.509 | ................................****.X | 271.337 | | 269.806 | ................................*****................*..*****..*...****X | 447.009 | | 308.643 | .....................................******.........................****X | 500.882 | | 427.271 | ....................................................X | 427.271 | | 409.063 | ..................................................*****..............****X | 516.132 | | 515.843 | ...............................................................X | 515.843 | | 507.863 | ..............................................................X | 507.863 | | 493.764 | .............................................................***X | 493.764 | | 430.924 | .....................................................X | 430.924 | | 398.325 | .................................................*****.*...****X | 429.800 | | 417.279 | ...................................................******...........****X | 504.141 | | 501.731 | ..............................................................X | 501.731 | | 490.796 | ............................................................*****........*****X | 548.893 | | 507.594 | ..............................................................*****...*.....****X | 565.239 | | 533.587 | ..................................................................*.....****X | 572.334 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 14:56:16,134 | test | WARNING | Thread-100362 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:pv80sO-22-468000:1%FSJ5Z1N_8lccBZRHG-22-599000 create failed keys from task: [u'test_collections-370'] 2021-05-17 14:56:16,135 | test | WARNING | Thread-100366 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wY3QB1lE9C-37-311000:vp5mG_MiID1O1XU8_34h-37-470000 update failed keys from task: [u'test_collections-1150', u'test_collections-1074', u'test_collections-1068'] 2021-05-17 14:56:16,135 | test | WARNING | Thread-100364 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:wxdH8TjwVXb-23-106000:c1wUCbhm-23-199000 create failed keys from task: [u'test_collections-370'] 2021-05-17 14:56:16,157 | test | WARNING | Thread-100430 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:2GXKs_zVV_P0YoE-22-813000:KPOljXF31in-22-961000 create failed keys from task: [u'test_collections-370'] 2021-05-17 14:56:16,157 | test | WARNING | Thread-100433 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 update failed keys from task: [u'test_collections-1848', u'test_collections-1926', u'test_collections-1807', u'test_collections-1929', u'test_collections-1941', u'test_collections-1842', u'test_collections-1888', u'test_collections-1802', u'test_collections-1847', u'test_collections-1882', u'test_collections-1918', u'test_collections-1854', u'test_collections-1833', u'test_collections-1912'] 2021-05-17 14:56:16,203 | test | WARNING | Thread-100373 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:q-15-684000:H6X-24-825000 create failed keys from task: [u'test_collections-202', u'test_collections-392'] 2021-05-17 14:56:16,220 | test | WARNING | Thread-100611 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:lOiYeU73--15-686000 delete failed keys from task: [u'test_collections-936'] 2021-05-17 14:56:16,226 | test | WARNING | Thread-100642 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:ZBc3ougCBv11YHy0EY-39-849000:Sp-41dhqDRf-39-977000 update failed keys from task: [u'test_collections-1622', u'test_collections-1659'] 2021-05-17 14:56:16,227 | test | WARNING | Thread-100647 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:sz3oAGnsT0UK95z-15-698000:WVZLtpYQtDT-15-700000 update failed keys from task: [u'test_collections-1258', u'test_collections-1401', u'test_collections-1226', u'test_collections-1325', u'test_collections-1308', u'test_collections-1418', u'test_collections-1429', u'test_collections-1183', u'test_collections-1263', u'test_collections-1298'] 2021-05-17 14:56:16,220 | test | WARNING | Thread-100612 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DK5cIbCBLiu2RnJ%0-15-689000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 14:56:16,243 | test | WARNING | Thread-100618 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:m%XWgVMc2hPIX-15-688000 update failed keys from task: [u'test_collections-1423', u'test_collections-1325', u'test_collections-1417', u'test_collections-1183', u'test_collections-1263', u'test_collections-1244', u'test_collections-1201', u'test_collections-1223'] 2021-05-17 14:56:16,226 | test | WARNING | Thread-100643 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:ZBc3ougCBv11YHy0EY-39-849000:-NfIU-39-976000 update failed keys from task: [u'test_collections-1138', u'test_collections-1022'] 2021-05-17 14:56:16,259 | test | WARNING | Thread-100642 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:ZBc3ougCBv11YHy0EY-39-849000:Sp-41dhqDRf-39-977000 delete failed keys from task: [u'test_collections-759', u'test_collections-637', u'test_collections-756', u'test_collections-402', u'test_collections-672', u'test_collections-592'] 2021-05-17 14:56:16,273 | test | WARNING | Thread-100655 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:sz3oAGnsT0UK95z-15-698000:4uwrqog-15-699000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 14:56:16,273 | test | WARNING | Thread-100660 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:sz3oAGnsT0UK95z-15-698000:WXXSVZsOcnpq9EF-15-700000 update failed keys from task: [u'test_collections-1093', u'test_collections-988', u'test_collections-1103', u'test_collections-1019', u'test_collections-1132', u'test_collections-1177', u'test_collections-1124'] 2021-05-17 14:56:16,273 | test | WARNING | Thread-100659 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:sz3oAGnsT0UK95z-15-698000:5B-15-698000 update failed keys from task: [u'test_collections-1258', u'test_collections-1412', u'test_collections-1325'] 2021-05-17 14:56:16,293 | test | WARNING | Thread-100643 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:ZBc3ougCBv11YHy0EY-39-849000:-NfIU-39-976000 delete failed keys from task: [u'test_collections-99', u'test_collections-45', u'test_collections-193'] 2021-05-17 14:56:16,657 | test | WARNING | Thread-100433 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 delete failed keys from task: [u'test_collections-902', u'test_collections-925', u'test_collections-947', u'test_collections-801', u'test_collections-844', u'test_collections-960', u'test_collections-939', u'test_collections-835', u'test_collections-951', u'test_collections-991', u'test_collections-892'] 2021-05-17 14:56:16,677 | test | WARNING | Thread-100433 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Yyu_H-35-642000:vXKzlk%GhIp4VV-35-826000 create failed keys from task: [u'test_collections-5630', u'test_collections-5760'] 2021-05-17 14:56:30,319 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 14:57:49,516 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 14:57:49,516 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 14:58:12,536 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 97% 2021-05-17 14:58:12,536 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 98% 2021-05-17 14:58:28,601 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 14:58:28,602 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 14:59:27,960 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1400001049 seconds to complete 2021-05-17 14:59:27,973 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3455.456s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 28 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_28 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_29 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=full,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_29', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 29, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 14:59:37,486 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #29 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 14:59:38,173 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #29 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 14:59:38,598 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x10aefab type=client open_count=1 channel=[id: 0x1fd9279a, 0.0.0.0/0.0.0.0:54077] timeout=300.0>) 2021-05-17 14:59:59,352 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x10aefac type=client open_count=1 channel=[id: 0x030b924c, 0.0.0.0/0.0.0.0:48490] timeout=300.0>) 2021-05-17 15:00:01,371 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x10aefad type=client open_count=1 channel=[id: 0x532c1aa6, 0.0.0.0/0.0.0.0:49844] timeout=300.0>) 2021-05-17 15:00:03,387 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10aefae type=client open_count=1 channel=[id: 0xc058c56a, 0.0.0.0/0.0.0.0:49529] timeout=300.0>) 2021-05-17 15:00:27,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10aefaf type=client open_count=1 channel=[id: 0xfa682b9e, 0.0.0.0/0.0.0.0:36861] timeout=300.0>) 2021-05-17 15:00:29,236 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10aefb0 type=client open_count=1 channel=[id: 0x219b685c, 0.0.0.0/0.0.0.0:50169] timeout=300.0>) 2021-05-17 15:00:31,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x10aefb1 type=client open_count=1 channel=[id: 0x45938cf6, 0.0.0.0/0.0.0.0:39247] timeout=300.0>) 2021-05-17 15:00:55,194 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x10aefb2 type=client open_count=1 channel=[id: 0x3667c6ec, 0.0.0.0/0.0.0.0:41500] timeout=300.0>) 2021-05-17 15:00:57,213 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x10aefb3 type=client open_count=1 channel=[id: 0xb933dbfe, 0.0.0.0/0.0.0.0:40756] timeout=300.0>) 2021-05-17 15:00:59,230 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x10aefb4 type=client open_count=1 channel=[id: 0x73dbb7d1, 0.0.0.0/0.0.0.0:37049] timeout=300.0>) 2021-05-17 15:01:23,250 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x10aefb5 type=client open_count=1 channel=[id: 0xfd63aaaf, 0.0.0.0/0.0.0.0:48720] timeout=300.0>) 2021-05-17 15:01:25,267 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x10aefb6 type=client open_count=1 channel=[id: 0xf2b4172c, 0.0.0.0/0.0.0.0:38630] timeout=300.0>) 2021-05-17 15:01:27,288 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x10aefb7 type=client open_count=1 channel=[id: 0x3e3905ca, 0.0.0.0/0.0.0.0:33161] timeout=300.0>) 2021-05-17 15:01:51,275 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x10aefb8 type=client open_count=1 channel=[id: 0x2d67ad9f, 0.0.0.0/0.0.0.0:46953] timeout=300.0>) 2021-05-17 15:01:53,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x10aefb9 type=client open_count=1 channel=[id: 0xf0ba65c8, 0.0.0.0/0.0.0.0:33872] timeout=300.0>) 2021-05-17 15:01:55,312 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 15:02:12,207 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 15:02:25,851 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 15:02:25,872 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #29 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 15:02:25,914 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #29 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 15:02:39,875 | test | INFO | pool-31-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.56351150891 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 15:02:39,901 | test | INFO | pool-31-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 15:02:49,992 | test | INFO | pool-31-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 15:02:50,017 | test | INFO | pool-31-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1419999599 sec 2021-05-17 15:02:50,275 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 15:03:38,265 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 15:03:38,579 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 714504520 | 82777940 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 675019744 | 107455074 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 858544488 | 56421717 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 15:03:38,581 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 15:04:24,710 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 15:04:24,726 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 15:21:34,332 | test | INFO | pool-31-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******....................*................................................................................****X | 61.461 | | 49.167 | ..............................................................................X | 49.167 | | 0.000 | ***************************************..****X | 1.501 | | 0.000 | ***************************************************************..******..****X | 3.001 | | 2.401 | ..X | 2.401 | | 0.000 | *****..****X | 1.501 | | 1.201 | X | 1.201 | | 0.000 | ********************************************..******..*****X | 3.001 | | 3.001 | ...****X | 3.001 | | 0.000 | **********************************.*......****X | 4.500 | | 1.501 | .****X | 1.501 | | 1.500 | .X | 1.500 | | 1.500 | .*****...................*****X | 13.502 | | 10.702 | ................X | 10.702 | | 7.501 | ...........***X | 7.501 | | 4.701 | ......X | 4.701 | | 1.500 | .*****X | 1.500 | | 1.500 | .*****...........*....****X | 10.501 | | 0.000 | ***********..*..*****..........****X | 9.002 | | 8.401 | ............X | 8.401 | | 6.702 | .........****X | 6.702 | | 5.802 | ........X | 5.802 | | 0.800 | ****X | 0.800 | | 0.500 | X | 0.500 | | 0.000 | *****.*...*****.....****X | 5.600 | | 5.100 | .......X | 5.100 | | 4.900 | ......****X | 4.900 | | 0.000 | ***********.*.....*****......****X | 7.702 | | 4.702 | ......X | 4.702 | | 1.500 | .****X | 1.500 | | 0.000 | ******..*****......*........***X | 10.000 | | 8.601 | ............X | 8.601 | | 8.000 | ............****X | 8.000 | | 6.300 | .........X | 6.300 | | 4.501 | ......***X | 4.501 | | 2.401 | ..X | 2.401 | | 0.000 | *****..........*****X | 6.303 | | 5.700 | ........****X | 5.700 | | 5.401 | .......X | 5.401 | | 4.500 | ......****X | 4.500 | | 0.000 | ******.......******......*..****.*****X | 9.901 | | 1.500 | .**********X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | **********...............................*****X | 19.500 | | 6.001 | ........****X | 6.001 | | 5.998 | ........*****X | 5.998 | | 1.600 | .X | 1.600 | | 0.000 | *****....****X | 2.999 | | 2.699 | ...X | 2.699 | | 1.500 | .*****..*****X | 3.000 | | 3.000 | ...*********X | 3.000 | | 0.800 | X | 0.800 | | 0.000 | *****..*..***X | 3.000 | | 3.000 | ...X | 3.000 | | 3.000 | ...*****..*.*****..*..........****X | 12.000 | | 0.000 | *************************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 15:21:34,869 | test | INFO | pool-31-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******...............................................................*...............*****......................****X | 831.903 | | 589.841 | .....................................................................X | 589.841 | | 522.187 | .............................................................*****.........*****.....*.......****X | 691.511 | | 620.100 | .........................................................................*****....*..............****X | 769.083 | | 636.213 | ...........................................................................X | 636.213 | | 580.279 | ....................................................................****X | 580.279 | | 528.280 | ..............................................................X | 528.280 | | 497.749 | ..........................................................***X | 497.749 | | 480.639 | ........................................................X | 480.639 | | 443.612 | ....................................................*****....****X | 477.011 | | 468.201 | .......................................................X | 468.201 | | 458.290 | ......................................................*****.....*..****X | 521.627 | | 521.427 | .............................................................X | 521.427 | | 514.354 | ............................................................****X | 514.354 | | 474.543 | ........................................................*****....*..****X | 528.247 | | 524.306 | ..............................................................*****X | 524.306 | | 468.520 | .......................................................******...........*...***X | 585.146 | | 581.211 | ....................................................................*.****X | 586.650 | | 575.762 | ....................................................................X | 575.762 | | 531.839 | ..............................................................****X | 531.839 | | 445.355 | ....................................................*****.*...****X | 479.795 | | 473.764 | .......................................................****X | 473.764 | | 466.937 | .......................................................*****X | 470.808 | | 452.417 | .....................................................X | 452.417 | | 401.565 | ...............................................*****.............*...***X | 537.739 | | 524.372 | ..............................................................X | 524.372 | | 504.931 | ...........................................................*****.*.***X | 516.005 | | 459.788 | ......................................................*****..*.....***X | 519.865 | | 498.091 | ..........................................................X | 498.091 | | 408.892 | ................................................*****.........******...****X | 509.785 | | 504.827 | ...........................................................X | 504.827 | | 489.771 | .........................................................****X | 489.771 | | 422.478 | .................................................*****....*.......****X | 510.156 | | 484.639 | .........................................................*****..X | 500.618 | | 493.914 | ..........................................................*****........*****X | 564.352 | | 559.438 | ..................................................................X | 559.438 | | 549.930 | .................................................................***X | 549.930 | | 485.865 | .........................................................X | 485.865 | | 466.716 | .......................................................*****...****X | 493.849 | | 472.549 | .......................................................X | 472.549 | | 427.658 | ..................................................****X | 427.658 | | 392.053 | ..............................................*****...*...****X | 446.047 | | 428.562 | ..................................................******......****X | 481.928 | | 430.134 | ..................................................X | 430.134 | | 386.127 | .............................................*****..................****X | 536.372 | | 499.561 | ...........................................................X | 499.561 | | 481.865 | ........................................................****X | 481.865 | | 440.832 | ...................................................******......****X | 487.536 | | 422.748 | .................................................X | 422.748 | | 414.050 | ................................................*****.X | 421.923 | | 419.513 | .................................................****....*..........****X | 533.276 | | 519.805 | .............................................................X | 519.805 | | 511.690 | ............................................................****X | 511.690 | | 510.085 | ............................................................X | 510.085 | | 475.462 | ........................................................****X | 475.462 | | 421.214 | .................................................******...*****.X | 451.929 | | 438.212 | ...................................................****X | 438.212 | | 419.207 | .................................................******.............****X | 524.881 | | 484.635 | .........................................................X | 484.635 | | 475.710 | ........................................................*****..*****X | 493.871 | | 447.908 | ....................................................****X | 447.908 | | 432.079 | ..................................................****X | 432.079 | | 413.588 | ................................................X | 413.588 | | 410.662 | ................................................****X | 410.662 | | 366.002 | ..........................................******..........****X | 444.622 | | 432.289 | ..................................................*.*****........X | 500.498 | | 497.005 | ..........................................................****X | 497.005 | | 471.504 | .......................................................*****X | 471.504 | | 412.402 | ................................................X | 412.402 | | 379.899 | ............................................***X | 379.899 | | 373.494 | ...........................................X | 373.494 | | 331.472 | ......................................X | 331.472 | | 324.358 | .....................................****................*...****X | 476.138 | | 407.315 | ...............................................****X | 407.315 | | 403.708 | ...............................................X | 403.708 | | 402.355 | ...............................................*****.............*****X | 514.308 | | 441.447 | ....................................................X | 441.447 | | 440.299 | ...................................................****.......*.****X | 504.711 | | 500.802 | ...........................................................X | 500.802 | | 422.598 | .................................................*****.......******..****X | 498.290 | | 486.515 | .........................................................X | 486.515 | | 485.957 | .........................................................****X | 485.957 | | 467.545 | .......................................................*.*****...****X | 504.773 | | 497.372 | ..........................................................*.****X | 500.000 | | 424.067 | .................................................****X | 424.067 | | 422.966 | .................................................*.*****......*....****.*****X | 517.721 | | 513.741 | ............................................................X | 513.741 | | 465.958 | .......................................................*****..****X | 484.806 | | 482.424 | ........................................................*...X | 499.816 | | 465.501 | ......................................................***X | 465.501 | | 463.263 | ......................................................*...****X | 488.442 | | 451.799 | .....................................................*****..*.*****......****X | 525.066 | | 453.710 | .....................................................X | 453.710 | | 432.991 | ...................................................*****.......X | 496.048 | | 495.667 | ..........................................................*****..*........***X | 575.841 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:21:35,895 | test | INFO | pool-31-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******.........................................................................*...........................****X | 124.421 | | 0.000 | ***************************************.*..*****.....****X | 10.510 | | 3.900 | ..X | 3.900 | | 1.501 | ****X | 1.501 | | 0.301 | *****.*.*.***X | 4.202 | | 2.500 | .X | 2.500 | | 1.501 | ****X | 1.501 | | 1.501 | ******X | 1.501 | | 1.501 | ***X | 1.501 | | 1.201 | X | 1.201 | | 0.000 | ***********************.****X | 1.501 | | 1.501 | *****X | 1.501 | | 0.000 | ******.*****...*****X | 6.002 | | 0.000 | *******.****.*.****X | 4.501 | | 2.401 | X | 2.401 | | 0.000 | **********.*****.*.***X | 4.502 | | 3.802 | ..X | 3.802 | | 3.001 | .****X | 3.001 | | 2.701 | .X | 2.701 | | 2.201 | ****X | 2.201 | | 0.800 | *****X | 0.800 | | 0.000 | ******.*********X | 1.501 | | 1.101 | X | 1.101 | | 0.000 | ******.*****.******..*...***X | 9.201 | | 5.002 | ...X | 5.002 | | 0.000 | ******..*****.****X | 4.501 | | 4.201 | ..X | 4.201 | | 3.000 | .*********X | 3.000 | | 3.000 | .*X | 3.000 | | 3.000 | .****..*..****X | 7.500 | | 6.000 | ...X | 6.000 | | 0.000 | ***********..*****.******..*.****X | 7.502 | | 4.002 | ..X | 4.002 | | 0.000 | ***********.*****.******.****X | 4.499 | | 0.000 | ************...*****.*****X | 6.001 | | 0.000 | *****************.*.***X | 3.000 | | 0.800 | X | 0.800 | | 0.000 | ****************.*.*****...****X | 7.100 | | 5.101 | ...X | 5.101 | | 0.400 | *****.*******.*****.****X | 4.201 | | 3.500 | .X | 3.500 | | 3.300 | .****X | 3.300 | | 0.000 | ***********************...****X | 4.500 | | 2.100 | X | 2.100 | | 0.000 | ******.****X | 1.499 | | 0.000 | ******...****X | 4.499 | | 1.199 | X | 1.199 | | 0.000 | *****.*****.****X | 3.000 | | 0.800 | X | 0.800 | | 0.000 | ************.***X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ******.*****.****X | 3.000 | | 0.800 | X | 0.800 | | 0.000 | *************************************X | 0.000 | +-------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:21:36,176 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 71.3538107698 | 6437847040 | 4321112064 | 2422554624 / 7511588864 | 101413 / 202617 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 61.6933333333 | 6437847040 | 4286500864 | 2412773376 / 7511588864 | 98634 / 199860 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 52.5517517251 | 6437847040 | 4221669376 | 2477682688 / 7511588864 | 100747 / 200438 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 75.1304144931 | 6437847040 | 3791187968 | 3090644992 / 9530753024 | 101800 / 202400 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 77.7333333333 | 6437847040 | 4259598336 | 2453827584 / 7511588864 | 101600 / 203200 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 15:21:49,437 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 15:23:15,917 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164016792 | 249949912 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 161614176 | 258055999 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 442216472 | 432681340 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 15:23:15,918 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #29 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 15:23:26,002 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 15:23:26,003 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 15:23:26,003 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 15:23:36,959 | test | INFO | pool-31-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.97699999809 seconds to complete 2021-05-17 15:23:36,964 | test | INFO | pool-31-thread-23 | [common_lib:sleep:22] Sleep 8.97699999809 seconds. Reason: Wait after rebalance complete 2021-05-17 15:25:46,003 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 15:25:46,003 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 15:25:46,003 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 15:25:52,964 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 15:39:47,835 | test | INFO | pool-31-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..........*.....*****...................................******................******....*....****X | 142.203 | | 141.386 | ........................................................................X | 141.386 | | 135.505 | .....................................................................****X | 135.505 | | 128.236 | .................................................................*X | 128.236 | | 126.485 | ................................................................***X | 126.485 | | 113.946 | ..........................................................*****.*....*****.....*.............****X | 158.073 | | 134.511 | .....................................................................******.......*****X | 148.769 | | 132.819 | ....................................................................*****X | 132.819 | | 128.572 | .................................................................******..................................*****X | 192.058 | | 157.465 | ................................................................................X | 157.465 | | 136.068 | .....................................................................*****.........*........***X | 167.408 | | 163.408 | ....................................................................................X | 163.408 | | 136.847 | ......................................................................*****X | 138.247 | | 133.482 | ....................................................................****X | 133.482 | | 102.580 | ....................................................X | 102.580 | | 75.211 | ......................................*****......*.....****X | 97.299 | | 89.035 | .............................................X | 89.035 | | 86.745 | ............................................*****..*...........****X | 112.600 | | 106.703 | ......................................................X | 106.703 | | 92.722 | ...............................................****X | 92.722 | | 78.210 | .......................................X | 78.210 | | 69.942 | ...................................****X | 69.942 | | 68.042 | ..................................X | 68.042 | | 65.152 | ................................****X | 65.152 | | 55.246 | ...........................X | 55.246 | | 51.958 | ..........................*****.......*****X | 65.767 | | 65.563 | .................................************....*..****X | 77.458 | | 60.159 | ..............................*****X | 60.159 | | 60.059 | ..............................X | 60.059 | | 58.770 | .............................****X | 58.770 | | 43.777 | .....................******........*****X | 58.671 | | 40.385 | ....................*****X | 40.385 | | 30.484 | ..............X | 30.484 | | 0.500 | *****X | 0.500 | | 0.400 | ****X | 0.400 | | 0.000 | ************.************.*****X | 4.197 | | 0.300 | *****************.*********X | 3.398 | | 3.398 | **X | 3.398 | | 1.599 | ****X | 1.599 | | 0.700 | X | 0.700 | | 0.200 | ***********.*****X | 3.799 | | 1.999 | ******X | 2.299 | | 1.799 | ****X | 1.799 | | 0.000 | **********X | 1.200 | | 1.000 | X | 1.000 | | 0.600 | *****.******.*.*****X | 5.798 | | 1.700 | *****X | 1.700 | | 0.300 | ****X | 0.300 | | 0.200 | X | 0.200 | | 0.000 | ******.*****X | 2.199 | | 1.699 | *****.X | 1.999 | | 1.900 | *.***X | 2.300 | | 1.099 | *X | 1.099 | | 0.200 | *********X | 1.000 | | 0.500 | ************.*****X | 3.199 | | 3.099 | ****X | 3.099 | | 1.799 | X | 1.799 | | 0.000 | ******.**********X | 2.199 | | 1.398 | ******.******X | 2.797 | | 2.698 | ***********.*****X | 3.897 | | 3.096 | *****X | 3.096 | | 1.498 | ******.*****X | 2.896 | | 1.696 | *****X | 1.696 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:39:48,875 | test | INFO | pool-31-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******.....................................*..............*****...................................*..............****X | 799.277 | | 386.819 | ...............................................****X | 386.819 | | 380.410 | ..............................................X | 380.410 | | 373.909 | .............................................*****......*.*****.*....*****...................................*****X | 749.941 | | 703.581 | .......................................................................................*****X | 703.581 | | 649.187 | ................................................................................*****X | 649.187 | | 507.238 | ..............................................................*X | 507.238 | | 507.238 | ..............................................................***X | 507.238 | | 378.314 | ..............................................X | 378.314 | | 378.314 | ..............................................*X | 378.314 | | 378.314 | ..............................................*X | 378.314 | | 334.932 | ........................................X | 334.932 | | 307.183 | .....................................*****.....*....*****........*...*****.........................*****X | 670.123 | | 444.423 | ......................................................X | 444.423 | | 351.613 | ..........................................***X | 351.613 | | 314.017 | ......................................X | 314.017 | | 217.962 | ..........................******.*****.X | 232.644 | | 231.038 | ...........................*****.............*****X | 329.539 | | 155.705 | ..................******...................................................X | 566.273 | | 566.273 | .....................................................................****X | 566.273 | | 437.139 | .....................................................*****X | 437.139 | | 382.278 | ..............................................*****X | 382.278 | | 221.619 | ..........................******.............******......*****X | 371.295 | | 313.673 | ......................................X | 313.673 | | 299.316 | ....................................****X | 299.316 | | 181.192 | .....................******..******.******...........................*****X | 422.492 | | 247.662 | .............................******............*...........****X | 430.278 | | 292.687 | ...................................******.....******.......X | 383.691 | | 383.691 | ...............................................***X | 383.691 | | 383.691 | ...............................................*.*...........****X | 481.782 | | 426.719 | ....................................................******...........*****X | 514.643 | | 390.206 | ...............................................******.....****X | 431.424 | | 402.724 | .................................................X | 402.724 | | 329.765 | ........................................*****.*........****X | 402.449 | | 394.849 | ................................................X | 394.849 | | 348.311 | ..........................................*****...******...........*...........****X | 547.324 | | 540.239 | ..................................................................******...******....****X | 597.728 | | 562.843 | .....................................................................X | 562.843 | | 466.879 | .........................................................*****..*.*****.*.*****X | 509.133 | | 489.911 | ............................................................*****X | 494.175 | | 474.150 | ..........................................................****X | 474.150 | | 463.041 | ........................................................X | 463.041 | | 445.805 | ......................................................****X | 445.805 | | 442.472 | ......................................................*..****X | 463.235 | | 462.399 | ........................................................*****X | 462.399 | | 438.748 | .....................................................******..*****X | 450.917 | | 435.419 | .....................................................X | 435.419 | | 433.309 | .....................................................****X | 433.309 | | 431.376 | ....................................................X | 431.376 | | 418.386 | ...................................................*****..*****.*....X | 474.966 | | 463.497 | ........................................................***X | 463.497 | | 463.097 | ........................................................*.*****....X | 499.048 | | 497.523 | .............................................................*****..*....****X | 544.574 | | 513.071 | ...............................................................X | 513.071 | | 472.641 | ..........................................................****X | 472.641 | | 446.177 | ......................................................*..****X | 462.563 | | 449.315 | .......................................................X | 449.315 | | 431.401 | ....................................................*****.....*..........****X | 544.896 | | 525.196 | ................................................................X | 525.196 | | 472.629 | ..........................................................****X | 472.629 | | 397.761 | ................................................******.................*****X | 533.703 | | 501.105 | .............................................................****X | 501.105 | | 501.018 | .............................................................*.*****...*...****X | 551.529 | | 531.870 | .................................................................******........*****X | 592.357 | | 526.369 | ................................................................******.........******...******...X | 646.421 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:39:49,063 | test | INFO | pool-31-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ************.............................................*.............*****...............................****X | 98.105 | | 95.603 | ......................................................................................X | 95.603 | | 90.302 | .................................................................................*****.................*.****X | 109.715 | | 95.951 | ......................................................................................X | 95.951 | | 84.959 | ............................................................................*****.....................*****X | 107.559 | | 100.998 | ...........................................................................................*****X | 100.998 | | 99.287 | .........................................................................................*****...*****X | 102.707 | | 96.585 | .......................................................................................****X | 96.585 | | 93.970 | ....................................................................................X | 93.970 | | 88.765 | ...............................................................................*..***X | 90.684 | | 80.107 | ........................................................................X | 80.107 | | 73.939 | ..................................................................****X | 73.939 | | 59.329 | .....................................................X | 59.329 | | 56.242 | ..................................................*****.*.****X | 58.642 | | 57.141 | ...................................................X | 57.141 | | 54.752 | ................................................*****...*..........****X | 68.032 | | 65.732 | ..........................................................X | 65.732 | | 60.945 | ......................................................*****.*..*****............*......****X | 83.924 | | 76.860 | .....................................................................*****X | 76.860 | | 74.139 | ..................................................................X | 74.139 | | 72.744 | .................................................................*****...........*.....*****.............****X | 104.821 | | 99.921 | ..........................................................................................X | 99.921 | | 85.845 | .............................................................................****X | 85.845 | | 76.443 | ....................................................................X | 76.443 | | 68.755 | .............................................................*****.....X | 74.462 | | 74.369 | ..................................................................*****..*........****X | 85.452 | | 81.950 | .........................................................................X | 81.950 | | 69.663 | ..............................................................*****...*...*****.....X | 81.346 | | 78.755 | ......................................................................*****........****X | 87.059 | | 87.059 | ..............................................................................X | 87.059 | | 77.964 | ......................................................................X | 77.964 | | 65.172 | ..........................................................******...******............*****......X | 88.677 | | 88.266 | ...............................................................................****X | 88.266 | | 82.963 | ..........................................................................X | 82.963 | | 78.864 | ......................................................................****X | 78.864 | | 77.768 | .....................................................................X | 77.768 | | 73.372 | .................................................................*****X | 73.372 | | 49.472 | ............................................******......******.......*......****..*........****X | 81.970 | | 77.269 | .....................................................................X | 77.269 | | 69.374 | ..............................................................******..............*****X | 84.576 | | 68.984 | .............................................................******............*****X | 82.274 | | 27.292 | .......................X | 27.292 | | 5.397 | ...*X | 5.397 | | 5.397 | ...**X | 5.397 | | 0.800 | *****X | 0.800 | | 0.700 | ******.*****..*****X | 4.198 | | 1.700 | ******X | 1.799 | | 1.599 | ****X | 1.599 | | 0.700 | X | 0.700 | | 0.400 | ****X | 0.500 | | 0.300 | ******.******..*****X | 3.799 | | 3.699 | ..****X | 3.699 | | 2.000 | *****X | 2.000 | | 0.000 | ************.*****X | 1.699 | | 1.199 | X | 1.199 | | 1.099 | ******.*****X | 3.101 | | 0.600 | ******.****X | 1.500 | | 0.400 | ******..*************.***X | 3.299 | | 2.099 | *X | 2.099 | | 0.000 | *****.*****.*..****X | 5.399 | | 3.898 | ..X | 3.898 | | 0.998 | *****.*.******************.****X | 3.497 | | 2.796 | .*****X | 2.796 | | 1.997 | ******...*****X | 4.394 | | 1.197 | *****X | 1.197 | | 0.799 | X | 0.799 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:39:49,311 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 15:39:49,362 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 15:40:00,617 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 15:40:00,793 | test | INFO | pool-31-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.625 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 47.1092148464 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 43.670210993 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 62.8660477675 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 66.1455642406 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 15:40:00,864 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 15:40:05,963 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 15:40:08,581 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 15:40:11,049 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 15:40:16,118 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 15:40:21,250 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.12 2021-05-17 15:40:26,318 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 4.85 2021-05-17 15:40:27,561 | test | INFO | pool-31-thread-11 | [table_view:display:72] Keys failed in bucket1:_default:Wz-50-336000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:31,427 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 8.89 2021-05-17 15:40:32,882 | test | INFO | pool-31-thread-15 | [table_view:display:72] Keys failed in bucket1:8trU2oELD_w4NaqhD-50-688000:KHp1LLeNUA9oOJPcend-50-856000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1132 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:35,138 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in bucket1:scope2:mOdo-50-341000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:35,138 | test | INFO | pool-31-thread-19 | [table_view:display:72] Keys failed in bucket1:Zc_wT7XcbsKGqPPVMYr-50-343000:5%YuClzubKW-50-344000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:36,529 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 13.24 2021-05-17 15:40:38,117 | test | INFO | pool-31-thread-13 | [table_view:display:72] Keys failed in bucket1:8trU2oELD_w4NaqhD-50-688000:KHp1LLeNUA9oOJPcend-50-856000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-68 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-128 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-62 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-74 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-127 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-80 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:41,723 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.49 2021-05-17 15:40:46,250 | test | INFO | pool-31-thread-15 | [table_view:display:72] Keys failed in bucket1:8trU2oELD_w4NaqhD-50-688000:KHp1LLeNUA9oOJPcend-50-856000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1258 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1379 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1263 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:40:46,798 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 21.4 2021-05-17 15:40:51,911 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 25.21 2021-05-17 15:40:57,075 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 29.13 2021-05-17 15:41:02,164 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 32.61 2021-05-17 15:41:07,250 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.57 2021-05-17 15:41:12,342 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-17 15:41:17,482 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 38.26 2021-05-17 15:41:19,483 | test | INFO | pool-31-thread-21 | [table_view:display:72] Keys failed in bucket2:Mc%BMQz32uk-50-212000:aBtnTT5rph5cF3DzL7-50-391000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1879 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:19,694 | test | INFO | pool-31-thread-18 | [table_view:display:72] Keys failed in bucket2:_default:jr61dK-50-323000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:19,694 | test | INFO | pool-31-thread-6 | [table_view:display:72] Keys failed in bucket2:_default:nTTaR-50-322000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:21,947 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1115 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1085 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1068 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:22,598 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-59 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-68 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-96 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-127 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:22,601 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.6 2021-05-17 15:41:27,727 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.48 2021-05-17 15:41:32,848 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 47.21 2021-05-17 15:41:32,966 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1241 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:34,829 | test | INFO | pool-31-thread-25 | [table_view:display:72] Keys failed in bucket2:Mc%BMQz32uk-50-212000:DJ-50-392000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1848 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1879 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:35,108 | test | INFO | pool-31-thread-5 | [table_view:display:72] Keys failed in bucket2:Mc%BMQz32uk-50-212000:DJ-50-392000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-902 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-869 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-858 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-936 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-914 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-942 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-973 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-960 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-884 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-892 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:37,950 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 49.9 2021-05-17 15:41:41,831 | test | INFO | pool-31-thread-11 | [table_view:display:72] Keys failed in bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:TxzRPOVI3vLp-49-534000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1019 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:41,898 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-317 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-335 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-236 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-369 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-287 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-251 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:43,062 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 52.8 2021-05-17 15:41:47,354 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1588 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1457 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:48,174 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 55.35 2021-05-17 15:41:49,289 | test | INFO | pool-31-thread-24 | [table_view:display:72] Keys failed in bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:K7tH-49-533000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1045 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:50,885 | test | INFO | pool-31-thread-11 | [table_view:display:72] Keys failed in bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:TxzRPOVI3vLp-49-534000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1336 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1351 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1320 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:53,269 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 57.52 2021-05-17 15:41:54,437 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-529 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-454 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-411 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-581 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:41:58,358 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 60.05 2021-05-17 15:41:59,819 | test | INFO | pool-31-thread-24 | [table_view:display:72] Keys failed in bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:K7tH-49-533000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1258 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:01,907 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1659 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:03,460 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 62.3 2021-05-17 15:42:08,572 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 63.86 2021-05-17 15:42:13,686 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.66 2021-05-17 15:42:19,098 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 15:42:24,207 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 69.71 2021-05-17 15:42:27,937 | test | INFO | pool-31-thread-22 | [table_view:display:72] Keys failed in default:9ruw-52-133000:aX9cSx-52-250000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-526 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-447 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-541 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-563 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:28,079 | test | INFO | pool-31-thread-3 | [table_view:display:72] Keys failed in default:_default:Qc7eE-50-350000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1115 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1085 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1155 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:29,354 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 71.79 2021-05-17 15:42:32,243 | test | INFO | pool-31-thread-19 | [table_view:display:72] Keys failed in default:scope2:TIUQpwN5Nb0RBt-50-356000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1325 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1189 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1266 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1223 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:33,882 | test | INFO | pool-31-thread-18 | [table_view:display:72] Keys failed in default:9ruw-52-133000:aX9cSx-52-250000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1749 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1634 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1674 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:34,006 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:GW-50-361000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-988 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1138 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1146 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:34,450 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 73.78 2021-05-17 15:42:39,571 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 75.74 2021-05-17 15:42:40,598 | test | INFO | pool-31-thread-3 | [table_view:display:72] Keys failed in default:_default:Qc7eE-50-350000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1190 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1204 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:41,446 | test | INFO | pool-31-thread-22 | [table_view:display:72] Keys failed in default:9ruw-52-133000:aX9cSx-52-250000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-603 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-637 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-734 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-753 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-793 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-672 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:41,983 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:MJJ8tROno7IBabcmeqaH-50-361000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1109 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1099 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1000 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1022 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1143 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:44,670 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.77 2021-05-17 15:42:45,448 | test | INFO | pool-31-thread-19 | [table_view:display:72] Keys failed in default:scope2:TIUQpwN5Nb0RBt-50-356000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1394 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1430 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:47,525 | test | INFO | pool-31-thread-29 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:GW-50-361000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1258 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1226 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1307 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1373 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1297 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:48,028 | test | INFO | pool-31-thread-18 | [table_view:display:72] Keys failed in default:9ruw-52-133000:aX9cSx-52-250000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1865 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1811 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1944 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1923 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:49,763 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.83 2021-05-17 15:42:50,157 | test | INFO | pool-31-thread-6 | [table_view:display:72] Keys failed in default:_default:RxqZW4FreflZSBCiYip-50-351000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-529 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-621 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-566 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-523 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-664 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-650 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-683 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-695 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:53,154 | test | INFO | pool-31-thread-28 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:WJMmYj5C2H%%fy-50-362000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:53,154 | test | INFO | pool-31-thread-15 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:n-50-362000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1080 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-982 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1149 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1110 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:54,842 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.23 2021-05-17 15:42:54,934 | test | INFO | pool-31-thread-3 | [table_view:display:72] Keys failed in default:_default:Qc7eE-50-350000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1417 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:55,148 | test | INFO | pool-31-thread-22 | [table_view:display:72] Keys failed in default:9ruw-52-133000:aX9cSx-52-250000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-907 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-933 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-835 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-976 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:57,895 | test | INFO | pool-31-thread-30 | [table_view:display:72] Keys failed in default:spkfddGRN3G57xEtAD-50-360000:MJJ8tROno7IBabcmeqaH-50-361000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1308 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1298 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1201 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:42:59,950 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 83.08 2021-05-17 15:43:03,249 | test | INFO | pool-31-thread-6 | [table_view:display:72] Keys failed in default:_default:RxqZW4FreflZSBCiYip-50-351000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-884 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 15:43:05,049 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 84.54 2021-05-17 15:43:10,138 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 86.26 2021-05-17 15:43:15,236 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 88.25 2021-05-17 15:43:20,322 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 90.7 2021-05-17 15:43:25,440 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 92.78 2021-05-17 15:43:30,533 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 94.71 2021-05-17 15:43:35,648 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 96.33 2021-05-17 15:43:40,739 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 97.88 2021-05-17 15:43:45,832 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: running, progress: 99.74 2021-05-17 15:43:55,976 | test | INFO | pool-31-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 15:43:56,028 | test | INFO | pool-31-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 235.23300004 sec 2021-05-17 15:45:59,664 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 3 docs from 1000 to 1960 2021-05-17 15:45:59,664 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 14 docs from 0 to 1000 2021-05-17 15:45:59,665 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 15:45:59,667 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 22 docs from 1000 to 1960 2021-05-17 15:45:59,667 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 15:45:59,668 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 8 docs from 0 to 1000 2021-05-17 15:46:00,338 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 15:46:00,338 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 15:46:00,341 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 22 docs from 1000 to 1960 2021-05-17 15:46:00,341 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 15:46:00,341 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 23 docs from 0 to 1000 2021-05-17 15:46:00,342 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 15:46:00,344 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 14 docs from 1000 to 1960 2021-05-17 15:46:00,346 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 6 docs from 1000 to 1960 2021-05-17 15:46:00,348 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 9 docs from 1000 to 1960 2021-05-17 15:46:00,349 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 18 docs from 0 to 1000 2021-05-17 15:48:02,266 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 10 docs from 980 to 1440 2021-05-17 15:48:02,266 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 10 docs from 980 to 1440 2021-05-17 15:48:02,269 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 15 docs from 980 to 1440 2021-05-17 15:48:02,270 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 12 docs from 980 to 1440 2021-05-17 15:48:02,272 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-17 15:48:02,272 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 15:48:02,272 | test | ERROR | pool-31-thread-16 | [task:call:3809] Failed to load 6 docs from 980 to 1440 2021-05-17 15:55:58,687 | test | INFO | pool-31-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...****.........................*........................................................................**X | 227.449 | | 194.751 | ....................................................................................X | 194.751 | | 107.294 | ..............................................***X | 107.294 | | 81.598 | ..................................*****......****X | 93.998 | | 77.150 | ................................X | 77.150 | | 72.270 | ..............................***X | 72.270 | | 62.390 | ..........................****...*...........****.....****......****X | 119.413 | | 119.091 | ...................................................X | 119.091 | | 115.689 | .................................................****................................................****X | 223.448 | | 126.939 | ......................................................***X | 126.939 | | 118.142 | ..................................................X | 118.142 | | 110.789 | ...............................................****.*.***X | 115.914 | | 113.588 | ................................................*****..*.****.****X | 121.115 | | 106.099 | .............................................****................*..............***X | 174.095 | | 162.803 | ......................................................................X | 162.803 | | 141.607 | .............................................................**X | 141.607 | | 135.688 | ..........................................................X | 135.688 | | 130.969 | ........................................................*****X | 131.521 | | 130.826 | ........................................................****.X | 132.085 | | 126.978 | ......................................................*****..................*****..X | 171.991 | | 161.786 | ......................................................................***X | 161.786 | | 142.600 | .............................................................X | 142.600 | | 134.590 | ..........................................................*****....****X | 145.187 | | 102.504 | ............................................X | 102.504 | | 95.200 | ........................................***X | 95.200 | | 89.021 | ......................................*.....****X | 101.290 | | 95.617 | .........................................*****X | 95.617 | | 94.396 | ........................................****X | 94.396 | | 82.600 | ...................................X | 82.600 | | 70.702 | ..............................X | 70.702 | | 67.299 | ............................***X | 67.299 | | 64.499 | ...........................*.****X | 67.699 | | 60.109 | .........................*****X | 60.910 | | 60.300 | .........................****X | 60.300 | | 24.907 | .........X | 24.907 | | 0.900 | ****X | 0.900 | | 0.600 | *********X | 0.600 | | 0.500 | X | 0.500 | | 0.400 | ***************X | 1.800 | | 1.800 | X | 1.800 | | 1.600 | ****X | 1.600 | | 1.200 | X | 1.200 | | 1.100 | ****X | 1.700 | | 1.700 | X | 1.700 | | 1.700 | ***X | 1.700 | | 0.400 | *****************.*****......*...****...*.......*........***X | 64.507 | | 35.808 | ..............X | 35.808 | | 13.199 | ....****X | 13.199 | | 4.700 | .X | 4.700 | | 0.600 | ****.*.......*****....*.............****X | 57.403 | | 45.801 | ...................X | 45.801 | | 40.606 | ................****.*....****X | 50.799 | | 13.902 | .....****X | 13.902 | | 6.902 | ..X | 6.902 | | 2.400 | ****X | 2.400 | | 1.300 | X | 1.300 | | 0.600 | ***X | 0.600 | | 0.600 | X | 0.600 | | 0.600 | ****X | 0.600 | | 0.300 | X | 0.300 | | 0.200 | *********X | 1.200 | | 0.500 | X | 0.500 | | 0.400 | ******.****X | 2.600 | | 0.600 | *****..................*.....****X | 53.893 | | 28.104 | ...........****X | 28.104 | | 17.201 | ......X | 17.201 | | 0.400 | *****.....*...................****X | 56.395 | | 26.603 | ..........****X | 26.603 | | 11.604 | ....X | 11.604 | | 1.200 | *****X | 1.400 | | 1.300 | *********X | 1.500 | | 1.300 | X | 1.300 | | 0.300 | **********.*****X | 2.800 | | 1.600 | **********X | 1.700 | | 0.600 | ******.****X | 2.800 | | 2.300 | X | 2.300 | | 2.200 | ****X | 2.200 | | 1.000 | ****X | 1.000 | | 0.600 | **********X | 1.400 | | 1.000 | *****X | 1.000 | | 1.000 | **********X | 1.400 | | 1.000 | X | 1.000 | | 0.600 | ***********.****X | 2.400 | | 1.400 | **********X | 1.600 | | 1.599 | *****X | 1.799 | | 0.400 | X | 0.400 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:55:58,943 | test | INFO | pool-31-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****...........................................****...............*........**X | 467.377 | | 284.437 | .......................................****X | 284.437 | | 246.036 | .................................X | 246.036 | | 232.585 | ................................**X | 232.585 | | 120.048 | ................*****.......*..**X | 186.443 | | 124.415 | ................X | 124.415 | | 105.788 | ..............****........*...........................***X | 352.883 | | 272.521 | .....................................*****......................................................****X | 649.292 | | 541.058 | ...........................................................................****X | 541.058 | | 281.240 | ......................................*X | 281.240 | | 281.240 | ......................................**X | 281.240 | | 253.480 | ..................................****...........*...........***X | 406.337 | | 284.140 | .......................................****X | 284.140 | | 206.688 | ............................*****.........****...*.....***X | 327.631 | | 216.987 | .............................****......*........***X | 309.951 | | 225.133 | ..............................****X | 225.133 | | 157.075 | .....................*****..............******...*..****...*..***X | 327.183 | | 327.183 | .............................................X | 327.183 | | 263.982 | ....................................X | 263.982 | | 263.982 | ....................................***X | 263.982 | | 232.707 | ................................X | 232.707 | | 213.082 | .............................*****....*****............*****....*......*****.........****X | 459.386 | | 289.527 | ........................................X | 289.527 | | 190.593 | ..........................*****..............*.........***X | 355.396 | | 325.509 | .............................................X | 325.509 | | 314.884 | ...........................................*****................*.........***X | 492.066 | | 416.150 | ..........................................................X | 416.150 | | 378.399 | ....................................................****X | 378.399 | | 325.396 | .............................................***********....*.....****X | 390.437 | | 318.010 | ............................................X | 318.010 | | 266.640 | ....................................*****........****X | 323.968 | | 323.441 | ............................................X | 323.441 | | 316.617 | ...........................................****.*..*****..******X | 351.601 | | 327.452 | .............................................X | 327.452 | | 327.452 | .............................................**X | 327.452 | | 279.049 | ......................................X | 279.049 | | 262.220 | ....................................*****................................*...................***X | 620.441 | | 609.018 | .....................................................................................X | 609.018 | | 600.435 | ....................................................................................*****...............****X | 704.356 | | 618.797 | ......................................................................................X | 618.797 | | 300.981 | .........................................****X | 300.981 | | 221.723 | ..............................*****..............*..............*.......**X | 469.463 | | 469.463 | .................................................................X | 469.463 | | 368.003 | ...................................................X | 368.003 | | 296.754 | .........................................****X | 296.754 | | 267.580 | ....................................X | 267.580 | | 251.962 | ..................................***X | 251.962 | | 237.857 | ................................X | 237.857 | | 158.700 | .....................*****.*..****X | 180.124 | | 157.800 | .....................*****.......*.....*..***X | 258.930 | | 224.818 | ..............................X | 224.818 | | 211.749 | .............................*****......*****.....*....****X | 319.296 | | 244.625 | .................................*****...............*.......****..*..*****......X | 471.141 | | 468.359 | .................................................................****X | 468.359 | | 334.717 | ..............................................****X | 334.717 | | 301.730 | .........................................*...*****.........X | 380.947 | | 379.255 | ....................................................****.....*..****X | 428.000 | | 249.679 | ..................................*****X | 249.679 | | 217.425 | .............................*****.................*.*****.................****X | 458.218 | | 378.708 | ....................................................*****....*.....*****.....*.****X | 480.600 | | 414.099 | .........................................................*****.X | 417.157 | | 408.266 | ........................................................*****.*****X | 413.400 | | 270.609 | .....................................*****.................****X | 390.791 | | 385.400 | .....................................................X | 385.400 | | 372.871 | ...................................................*****..............****X | 470.939 | | 451.420 | ...............................................................X | 451.420 | | 442.482 | .............................................................*****.........*........***X | 562.545 | | 548.769 | ............................................................................X | 548.769 | | 486.349 | ....................................................................****X | 486.349 | | 305.521 | ..........................................X | 305.521 | | 224.896 | ..............................*****............................*......*****.........*****X | 526.132 | | 337.668 | ..............................................*****..........*.....*****.....*.......****.*......*****...******...X | 619.212 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:55:59,918 | test | INFO | pool-31-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***......*.................****......................*.......................................................**X | 412.331 | | 55.007 | ............*****....**X | 74.010 | | 74.010 | ................*...*****.........*...***X | 132.203 | | 51.204 | ...........*****.**X | 57.312 | | 57.305 | ............X | 57.305 | | 52.992 | ...........****.....*****............*****.****X | 123.872 | | 116.203 | ...........................X | 116.203 | | 108.939 | .........................**X | 108.939 | | 89.835 | ....................X | 89.835 | | 80.487 | ..................****....*X | 98.510 | | 98.510 | ......................*X | 98.510 | | 91.912 | .....................X | 91.912 | | 71.495 | ................*****.......***X | 101.712 | | 99.414 | .......................X | 99.414 | | 86.308 | ...................****.***X | 88.199 | | 81.496 | ..................X | 81.496 | | 72.687 | ................****.*..****.........****X | 120.575 | | 107.623 | .........................*****X | 108.510 | | 106.483 | ........................*****...............*........***X | 199.794 | | 153.281 | ....................................X | 153.281 | | 86.997 | ....................*****....*****...****X | 115.994 | | 112.482 | ..........................X | 112.482 | | 96.000 | ......................****X | 96.000 | | 71.128 | ................X | 71.128 | | 58.597 | .............*****......*****.X | 88.217 | | 87.297 | ....................*****....****X | 104.399 | | 81.208 | ..................X | 81.208 | | 66.196 | ...............****X | 66.196 | | 61.205 | .............X | 61.205 | | 56.407 | ............***X | 56.407 | | 55.008 | ............X | 55.008 | | 49.900 | ...........*****......*...****.*........****X | 127.284 | | 122.890 | ............................X | 122.890 | | 116.409 | ...........................***X | 116.409 | | 112.296 | ..........................**********X | 113.523 | | 108.315 | .........................X | 108.315 | | 101.005 | .......................***X | 101.005 | | 97.802 | ......................X | 97.802 | | 94.298 | .....................****X | 94.298 | | 92.591 | .....................X | 92.591 | | 85.390 | ...................****X | 85.390 | | 62.009 | ..............****X | 62.009 | | 60.810 | .............X | 60.810 | | 60.506 | .............****X | 60.506 | | 13.307 | ..X | 13.307 | | 1.600 | ***X | 1.600 | | 1.600 | ******...*****.....*...*......***X | 70.296 | | 64.599 | ..............X | 64.599 | | 58.006 | .............*****....******...****X | 87.916 | | 84.602 | ...................X | 84.602 | | 84.388 | ...................****X | 84.388 | | 78.407 | ..................X | 78.407 | | 78.000 | .................*****.......*....****.*.****X | 129.401 | | 118.615 | ...........................****X | 118.615 | | 111.608 | ..........................*****X | 113.373 | | 73.715 | ................*****.*.*****.X | 83.308 | | 82.710 | ...................***X | 82.710 | | 67.008 | ...............X | 67.008 | | 2.100 | ****X | 2.100 | | 0.600 | ***********...............*....****X | 79.304 | | 17.895 | ...X | 17.895 | | 4.500 | ****X | 4.500 | | 0.400 | ***********.....*****.....*....****X | 58.005 | | 30.309 | ......X | 30.309 | | 22.799 | ....***X | 22.799 | | 18.901 | ...X | 18.901 | | 4.200 | ****X | 4.200 | | 1.400 | ****X | 1.400 | | 1.200 | X | 1.200 | | 0.200 | ***************X | 1.600 | | 1.000 | X | 1.000 | | 0.600 | ****X | 1.400 | | 1.400 | ****.......*..........****X | 74.203 | | 27.599 | .....****X | 27.599 | | 13.000 | ..X | 13.000 | | 4.799 | ****X | 4.799 | | 1.600 | *****X | 2.200 | | 1.800 | *********X | 1.999 | | 1.501 | X | 1.501 | | 1.401 | *****..............*****X | 60.299 | | 29.400 | ......X | 29.400 | | 21.502 | ....***X | 21.502 | | 17.302 | ...X | 17.302 | | 4.801 | X | 4.801 | | 0.800 | ****............*...****X | 62.094 | | 45.597 | ..........X | 45.597 | | 19.702 | ...***X | 19.702 | | 15.702 | ..X | 15.702 | | 1.400 | ****X | 1.400 | | 1.200 | *****X | 1.600 | | 1.400 | ******X | 1.801 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 15:55:59,970 | test | WARNING | Thread-104217 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:Wz-50-336000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 15:55:59,980 | test | WARNING | Thread-104254 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:Zc_wT7XcbsKGqPPVMYr-50-343000:5%YuClzubKW-50-344000 update failed keys from task: [u'test_collections-6'] 2021-05-17 15:56:00,000 | test | WARNING | Thread-104316 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8trU2oELD_w4NaqhD-50-688000:KHp1LLeNUA9oOJPcend-50-856000 update failed keys from task: [u'test_collections-1391', u'test_collections-1103', u'test_collections-1106', u'test_collections-1308', u'test_collections-1263', u'test_collections-1143', u'test_collections-1342', u'test_collections-1189', u'test_collections-1146', u'test_collections-1059', u'test_collections-1257', u'test_collections-1258', u'test_collections-1379', u'test_collections-1019', u'test_collections-1096', u'test_collections-1132', u'test_collections-1056', u'test_collections-1177', u'test_collections-1298', u'test_collections-1093', u'test_collections-1244', u'test_collections-1201'] 2021-05-17 15:56:00,005 | test | WARNING | Thread-104334 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:mOdo-50-341000 update failed keys from task: [u'test_collections-6'] 2021-05-17 15:56:00,009 | test | WARNING | Thread-104357 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:spkfddGRN3G57xEtAD-50-360000:MJJ8tROno7IBabcmeqaH-50-361000 update failed keys from task: [u'test_collections-1028', u'test_collections-1109', u'test_collections-1308', u'test_collections-1186', u'test_collections-1000', u'test_collections-1022', u'test_collections-1143', u'test_collections-1244', u'test_collections-1201', u'test_collections-1099', u'test_collections-1177', u'test_collections-1298'] 2021-05-17 15:56:00,012 | test | WARNING | Thread-104365 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:spkfddGRN3G57xEtAD-50-360000:n-50-362000 update failed keys from task: [u'test_collections-1080', u'test_collections-982', u'test_collections-1149', u'test_collections-1031', u'test_collections-1110', u'test_collections-1124'] 2021-05-17 15:56:00,015 | test | WARNING | Thread-104362 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:spkfddGRN3G57xEtAD-50-360000:GW-50-361000 update failed keys from task: [u'test_collections-988', u'test_collections-1226', u'test_collections-1106', u'test_collections-1109', u'test_collections-1307', u'test_collections-1342', u'test_collections-1146', u'test_collections-1258', u'test_collections-1138', u'test_collections-1336', u'test_collections-1096', u'test_collections-1373', u'test_collections-1099', u'test_collections-1297', u'test_collections-1212'] 2021-05-17 15:56:00,016 | test | WARNING | Thread-104371 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:spkfddGRN3G57xEtAD-50-360000:WJMmYj5C2H%%fy-50-362000 update failed keys from task: [u'test_collections-1080', u'test_collections-982', u'test_collections-1149', u'test_collections-1031', u'test_collections-1110', u'test_collections-1124'] 2021-05-17 15:56:00,029 | test | WARNING | Thread-104404 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Qc7eE-50-350000 update failed keys from task: [u'test_collections-1190', u'test_collections-1391', u'test_collections-1115', u'test_collections-1204', u'test_collections-1417', u'test_collections-1150', u'test_collections-1085', u'test_collections-1351', u'test_collections-1230', u'test_collections-1155'] 2021-05-17 15:56:00,030 | test | WARNING | Thread-104408 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:RxqZW4FreflZSBCiYip-50-351000 delete failed keys from task: [u'test_collections-529', u'test_collections-621', u'test_collections-566', u'test_collections-523', u'test_collections-664', u'test_collections-884', u'test_collections-683', u'test_collections-517', u'test_collections-518', u'test_collections-650', u'test_collections-695', u'test_collections-552', u'test_collections-592'] 2021-05-17 15:56:00,049 | test | WARNING | Thread-104458 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9ruw-52-133000:aX9cSx-52-250000 update failed keys from task: [u'test_collections-1749', u'test_collections-1820', u'test_collections-1634', u'test_collections-1854', u'test_collections-1865', u'test_collections-1811', u'test_collections-1944', u'test_collections-1923', u'test_collections-1674'] 2021-05-17 15:56:00,065 | test | WARNING | Thread-104528 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:nTTaR-50-322000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 15:56:00,068 | test | WARNING | Thread-104533 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:jr61dK-50-323000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 15:56:00,076 | test | WARNING | Thread-104561 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Mc%BMQz32uk-50-212000:DJ-50-392000 update failed keys from task: [u'test_collections-1848', u'test_collections-1952', u'test_collections-1879'] 2021-05-17 15:56:00,076 | test | WARNING | Thread-104563 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:TxzRPOVI3vLp-49-534000 update failed keys from task: [u'test_collections-1190', u'test_collections-1093', u'test_collections-1103', u'test_collections-1183', u'test_collections-1000', u'test_collections-1320', u'test_collections-1189', u'test_collections-1146', u'test_collections-1336', u'test_collections-1019', u'test_collections-1351', u'test_collections-1155', u'test_collections-1056', u'test_collections-1013'] 2021-05-17 15:56:00,079 | test | WARNING | Thread-104560 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Mc%BMQz32uk-50-212000:aBtnTT5rph5cF3DzL7-50-391000 update failed keys from task: [u'test_collections-1952', u'test_collections-1879'] 2021-05-17 15:56:00,079 | test | WARNING | Thread-104508 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:TIUQpwN5Nb0RBt-50-356000 update failed keys from task: [u'test_collections-1270', u'test_collections-1401', u'test_collections-1325', u'test_collections-1382', u'test_collections-1394', u'test_collections-1252', u'test_collections-1430', u'test_collections-1189', u'test_collections-1266', u'test_collections-1223'] 2021-05-17 15:56:00,119 | test | WARNING | Thread-104562 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:d0zBVz_iso4kOx9LyBTx-49-317000:K7tH-49-533000 update failed keys from task: [u'test_collections-1258', u'test_collections-1138', u'test_collections-1307', u'test_collections-1297', u'test_collections-1045', u'test_collections-1342'] 2021-05-17 15:56:00,118 | test | WARNING | Thread-104559 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 update failed keys from task: [u'test_collections-1588', u'test_collections-1226', u'test_collections-1106', u'test_collections-1143', u'test_collections-1068', u'test_collections-1387', u'test_collections-1059', u'test_collections-1457', u'test_collections-1217', u'test_collections-1659', u'test_collections-1418', u'test_collections-1172', u'test_collections-1096', u'test_collections-1252', u'test_collections-1444', u'test_collections-1764', u'test_collections-1204', u'test_collections-1085', u'test_collections-1241', u'test_collections-1115', u'test_collections-1150', u'test_collections-1394'] 2021-05-17 15:56:00,226 | test | WARNING | Thread-104561 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Mc%BMQz32uk-50-212000:DJ-50-392000 delete failed keys from task: [u'test_collections-907', u'test_collections-902', u'test_collections-869', u'test_collections-925', u'test_collections-942', u'test_collections-960', u'test_collections-884', u'test_collections-858', u'test_collections-936', u'test_collections-914', u'test_collections-951', u'test_collections-973', u'test_collections-897', u'test_collections-892'] 2021-05-17 15:56:01,066 | test | WARNING | Thread-104458 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9ruw-52-133000:aX9cSx-52-250000 delete failed keys from task: [u'test_collections-829', u'test_collections-907', u'test_collections-526', u'test_collections-603', u'test_collections-447', u'test_collections-541', u'test_collections-563', u'test_collections-933', u'test_collections-637', u'test_collections-835', u'test_collections-678', u'test_collections-734', u'test_collections-976', u'test_collections-753', u'test_collections-479', u'test_collections-897', u'test_collections-793', u'test_collections-672'] 2021-05-17 15:56:01,094 | test | WARNING | Thread-104316 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8trU2oELD_w4NaqhD-50-688000:KHp1LLeNUA9oOJPcend-50-856000 delete failed keys from task: [u'test_collections-68', u'test_collections-128', u'test_collections-62', u'test_collections-74', u'test_collections-127', u'test_collections-31', u'test_collections-80', u'test_collections-171'] 2021-05-17 15:56:01,244 | test | WARNING | Thread-104559 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:WisD%1AcF-49-766000:3LVKZExV5RIulEQK-49-910000 delete failed keys from task: [u'test_collections-59', u'test_collections-529', u'test_collections-96', u'test_collections-127', u'test_collections-369', u'test_collections-287', u'test_collections-162', u'test_collections-85', u'test_collections-214', u'test_collections-335', u'test_collections-454', u'test_collections-411', u'test_collections-251', u'test_collections-171', u'test_collections-225', u'test_collections-384', u'test_collections-581', u'test_collections-3', u'test_collections-317', u'test_collections-68', u'test_collections-236', u'test_collections-116', u'test_collections-153'] 2021-05-17 15:56:13,921 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 15:57:35,009 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 15:57:35,010 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 15:57:54,338 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 15:57:54,338 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 15:58:15,976 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 15:58:15,976 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 15:59:13,719 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.131000042 seconds to complete 2021-05-17 15:59:13,734 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3585.777s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 29 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_29 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_30 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=1,recovery_type=delta,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,step_count=1,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'recovery_type': 'delta', 'step_count': '1', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_30', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 30, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 15:59:23,403 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #30 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 15:59:24,079 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #30 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 15:59:24,747 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x114e4e7 type=client open_count=1 channel=[id: 0x2f09123e, 0.0.0.0/0.0.0.0:55433] timeout=300.0>) 2021-05-17 15:59:45,503 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x114e4e8 type=client open_count=1 channel=[id: 0xf4c21955, 0.0.0.0/0.0.0.0:44992] timeout=300.0>) 2021-05-17 15:59:47,539 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x114e4e9 type=client open_count=1 channel=[id: 0xb63899a9, 0.0.0.0/0.0.0.0:46255] timeout=300.0>) 2021-05-17 15:59:49,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x114e4ea type=client open_count=1 channel=[id: 0x72055b0c, 0.0.0.0/0.0.0.0:38223] timeout=300.0>) 2021-05-17 16:00:13,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x114e4eb type=client open_count=1 channel=[id: 0xdef687bf, 0.0.0.0/0.0.0.0:58235] timeout=300.0>) 2021-05-17 16:00:15,579 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x114e4ec type=client open_count=1 channel=[id: 0x652efb5f, 0.0.0.0/0.0.0.0:48627] timeout=300.0>) 2021-05-17 16:00:17,596 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x114e4ed type=client open_count=1 channel=[id: 0x01629295, 0.0.0.0/0.0.0.0:60089] timeout=300.0>) 2021-05-17 16:00:41,565 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x114e4ee type=client open_count=1 channel=[id: 0xdbedb587, 0.0.0.0/0.0.0.0:43280] timeout=300.0>) 2021-05-17 16:00:43,584 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x114e4ef type=client open_count=1 channel=[id: 0xeca90966, 0.0.0.0/0.0.0.0:43813] timeout=300.0>) 2021-05-17 16:00:45,602 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x114e4f0 type=client open_count=1 channel=[id: 0x89c02f0c, 0.0.0.0/0.0.0.0:49762] timeout=300.0>) 2021-05-17 16:01:09,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x114e4f1 type=client open_count=1 channel=[id: 0xba0d06c7, 0.0.0.0/0.0.0.0:48293] timeout=300.0>) 2021-05-17 16:01:11,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x114e4f2 type=client open_count=1 channel=[id: 0x0785cc01, 0.0.0.0/0.0.0.0:51703] timeout=300.0>) 2021-05-17 16:01:13,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x114e4f3 type=client open_count=1 channel=[id: 0x0cbb94e0, 0.0.0.0/0.0.0.0:48673] timeout=300.0>) 2021-05-17 16:01:37,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x114e4f4 type=client open_count=1 channel=[id: 0x672f17d6, 0.0.0.0/0.0.0.0:55923] timeout=300.0>) 2021-05-17 16:01:39,750 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x114e4f5 type=client open_count=1 channel=[id: 0x03eaddaf, 0.0.0.0/0.0.0.0:43669] timeout=300.0>) 2021-05-17 16:01:41,766 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 16:01:58,858 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 16:02:12,533 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 16:02:12,555 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #30 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 16:02:12,592 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #30 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 16:02:27,543 | test | INFO | pool-32-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.313338555643 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 16:02:27,568 | test | INFO | pool-32-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:02:37,654 | test | INFO | pool-32-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 16:02:37,690 | test | INFO | pool-32-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1460001469 sec 2021-05-17 16:02:38,036 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 16:03:31,394 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 16:03:31,668 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 583801160 | 92621105 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 601768224 | 106852366 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 667149784 | 66583208 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 16:03:31,670 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 16:04:17,127 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 16:04:17,141 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 16:21:03,555 | test | INFO | pool-32-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****.........................................................................*...........................***X | 181.379 | | 48.279 | .........................*X | 48.279 | | 0.000 | **************************************..*****.*******.****X | 7.398 | | 5.298 | .X | 5.298 | | 4.898 | .***X | 4.898 | | 2.798 | X | 2.798 | | 1.200 | *****.*****X | 2.999 | | 0.799 | X | 0.799 | | 0.000 | **************X | 1.500 | | 1.400 | X | 1.400 | | 0.000 | ***********.*.***X | 4.600 | | 4.299 | .X | 4.299 | | 0.000 | **********************...*.****X | 8.997 | | 4.598 | .X | 4.598 | | 2.999 | ****X | 2.999 | | 2.299 | X | 2.299 | | 1.600 | ****.*...****X | 7.398 | | 1.999 | X | 1.999 | | 0.000 | **************X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ****************X | 1.501 | | 1.500 | X | 1.500 | | 1.500 | *********X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | *********X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | *****...*.****X | 7.506 | | 1.501 | ******..****X | 5.204 | | 4.603 | .X | 4.603 | | 3.803 | .****X | 3.803 | | 1.099 | X | 1.099 | | 0.000 | ***************X | 1.501 | | 0.000 | *****..*.****X | 6.403 | | 3.100 | X | 3.100 | | 1.101 | ****X | 1.101 | | 0.501 | X | 0.501 | | 0.000 | *********************************.*****X | 3.001 | | 1.602 | X | 1.602 | | 0.000 | **********.*****X | 3.001 | | 0.000 | **********X | 1.500 | | 1.500 | *****X | 1.500 | | 0.000 | ************.****X | 3.001 | | 0.000 | ***********.****X | 3.001 | | 0.801 | X | 0.801 | | 0.000 | ***********X | 1.501 | | 1.500 | ***X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | ***************************************.****X | 3.001 | | 1.601 | X | 1.601 | | 0.000 | **************************X | 1.500 | | 1.200 | X | 1.200 | | 0.000 | ***********************************************************X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | ************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:21:03,967 | test | INFO | pool-32-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****....................................................................................................****X | 62.958 | | 16.758 | .........................*X | 16.758 | | 0.000 | **************************************....*..*****.........****X | 9.496 | | 3.496 | ....X | 3.496 | | 1.799 | .*****.......*..***X | 7.197 | | 6.897 | .........X | 6.897 | | 2.999 | ...*****....*.****X | 5.998 | | 1.598 | .X | 1.598 | | 0.000 | **********....*****X | 2.600 | | 0.000 | *******.****..*****X | 2.300 | | 0.000 | ******.......****X | 4.499 | | 3.399 | ....X | 3.399 | | 2.999 | ...****X | 2.999 | | 1.599 | .X | 1.599 | | 0.000 | *****.....*****X | 3.299 | | 2.699 | ...*****.....*...*****...........****X | 14.896 | | 10.596 | ...............X | 10.596 | | 8.996 | .............***X | 8.996 | | 2.398 | ..X | 2.398 | | 0.000 | **********.....*..****X | 4.499 | | 2.399 | ..X | 2.399 | | 0.000 | ***************************..*..****X | 2.999 | | 0.000 | *****.*..................*****X | 12.009 | | 1.501 | .****X | 1.501 | | 0.499 | X | 0.499 | | 0.000 | *****..*****X | 1.501 | | 1.501 | .****X | 1.501 | | 1.501 | .******....****X | 4.203 | | 3.303 | ....X | 3.303 | | 0.000 | *****....................*****X | 12.607 | | 11.406 | .................****X | 11.406 | | 9.006 | .............X | 9.006 | | 0.000 | *****...........*****X | 7.504 | | 0.000 | *****.*......****X | 4.902 | | 1.101 | ******.******.......****X | 6.003 | | 3.903 | .....X | 3.903 | | 1.501 | .*****..****X | 2.902 | | 2.302 | ..X | 2.302 | | 1.600 | .****X | 1.600 | | 1.300 | .X | 1.300 | | 1.201 | *****...******....****X | 5.101 | | 3.003 | ...X | 3.003 | | 2.001 | ..****X | 2.001 | | 2.001 | ..X | 2.001 | | 1.000 | ***X | 1.000 | | 0.300 | X | 0.300 | | 0.000 | ****************************..*****X | 1.501 | | 1.500 | .**********X | 1.500 | | 0.000 | *****.*.*****........*****...X | 8.202 | | 8.201 | ............****X | 8.201 | | 0.000 | ******...*....***X | 4.502 | | 3.802 | .....X | 3.802 | | 3.001 | ...****X | 3.001 | | 0.000 | ******..****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | **********..*..***X | 3.000 | | 3.000 | ...*..*****..............****X | 12.703 | | 10.602 | ...............X | 10.602 | | 3.000 | ...***X | 3.000 | | 1.600 | .X | 1.600 | | 0.000 | ***********..****X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ***********..******...*.***X | 4.200 | | 1.500 | .****X | 1.500 | | 0.800 | X | 0.800 | | 0.000 | *************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 16:21:05,040 | test | INFO | pool-32-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****............................................*****..............................................*..........****X | 803.946 | | 666.097 | .................................................................................****X | 666.097 | | 566.071 | .....................................................................X | 566.071 | | 551.089 | ...................................................................*****.....*****X | 586.957 | | 518.177 | ...............................................................X | 518.177 | | 510.736 | ..............................................................****...........X | 596.406 | | 594.698 | ........................................................................****X | 594.698 | | 502.169 | .............................................................*****.*.****X | 517.552 | | 476.743 | ..........................................................*****....X | 507.143 | | 456.909 | .......................................................****X | 456.909 | | 442.182 | ......................................................X | 442.182 | | 438.000 | .....................................................X | 438.000 | | 437.366 | .....................................................****......................*......***X | 666.046 | | 649.031 | ...............................................................................X | 649.031 | | 535.705 | .................................................................****X | 535.705 | | 534.742 | .................................................................X | 534.742 | | 528.350 | ................................................................****X | 528.350 | | 461.558 | ........................................................****X | 461.558 | | 401.161 | ................................................X | 401.161 | | 399.188 | ................................................*****............*...*****..***********X | 536.856 | | 532.187 | .................................................................****X | 532.187 | | 510.435 | ..............................................................*..****X | 524.424 | | 411.905 | ..................................................*****..*.*****..X | 455.599 | | 441.902 | .....................................................****.*.....*****.......*.****X | 549.947 | | 516.801 | ...............................................................****X | 516.801 | | 494.732 | ............................................................X | 494.732 | | 475.028 | ..........................................................*****..........****X | 558.468 | | 498.169 | ............................................................X | 498.169 | | 480.392 | ..........................................................****X | 480.392 | | 465.260 | ........................................................******..X | 482.035 | | 468.589 | .........................................................****X | 468.589 | | 457.524 | .......................................................*****.......*..****X | 527.493 | | 479.001 | ..........................................................*****....*..****X | 529.981 | | 423.502 | ...................................................******..........****X | 500.685 | | 475.970 | ..........................................................X | 475.970 | | 464.686 | ........................................................****X | 464.686 | | 458.278 | ........................................................*.*****......*****X | 519.675 | | 461.714 | ........................................................****X | 461.714 | | 435.325 | .....................................................X | 435.325 | | 430.835 | ....................................................*****.*.......****X | 491.006 | | 416.896 | ..................................................*****...............*****X | 530.948 | | 443.612 | ......................................................******...........****X | 536.346 | | 525.046 | ................................................................X | 525.046 | | 513.505 | ..............................................................****X | 513.505 | | 454.612 | .......................................................*****.*....****X | 497.100 | | 459.609 | ........................................................*****..*.*****........*****X | 549.203 | | 495.752 | ............................................................X | 495.752 | | 473.916 | .........................................................****..*.......****X | 540.175 | | 468.283 | .........................................................X | 468.283 | | 443.637 | ......................................................***X | 443.637 | | 439.519 | .....................................................*.*.****.X | 464.555 | | 461.526 | ........................................................****X | 461.526 | | 440.408 | .....................................................X | 440.408 | | 434.474 | .....................................................*****.........****X | 513.819 | | 502.732 | .............................................................X | 502.732 | | 476.078 | ..........................................................*****.X | 489.447 | | 478.801 | ..........................................................****X | 478.801 | | 452.472 | .......................................................*****...*..........****X | 562.224 | | 537.512 | .................................................................X | 537.512 | | 506.646 | ..............................................................****X | 506.646 | | 498.871 | .............................................................**********X | 503.375 | | 451.249 | .......................................................X | 451.249 | | 419.287 | ...................................................****.*....*****....X | 494.270 | | 492.605 | ............................................................****X | 492.605 | | 461.669 | ........................................................*****.....*....****X | 535.852 | | 489.890 | ...........................................................****X | 489.890 | | 479.941 | ..........................................................X | 479.941 | | 469.384 | .........................................................*****.....****X | 507.198 | | 482.858 | ...........................................................****X | 482.858 | | 472.865 | .........................................................X | 472.865 | | 461.457 | ........................................................X | 461.457 | | 450.107 | ......................................................****.......*....****X | 535.180 | | 534.488 | .................................................................*****X | 535.185 | | 530.865 | .................................................................****.X | 541.654 | | 519.979 | ...............................................................*****.....*..****X | 573.916 | | 473.943 | .........................................................****X | 473.943 | | 446.625 | ......................................................X | 446.625 | | 416.360 | ..................................................****X | 416.360 | | 410.587 | ..................................................*****.*....*****.X | 466.152 | | 446.407 | ......................................................*****...*****..****X | 483.507 | | 477.591 | ..........................................................*..*****.*.......****X | 556.630 | | 450.070 | ......................................................*****.....*.....***X | 523.691 | | 520.732 | ...............................................................X | 520.732 | | 503.160 | .............................................................***X | 503.160 | | 445.527 | ......................................................X | 445.527 | | 400.925 | ................................................*****...............*****..........*.......****X | 655.253 | | 648.828 | ...............................................................................X | 648.828 | | 608.056 | ..........................................................................***X | 608.056 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:21:05,331 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 71.8923686225 | 6437847040 | 4292784128 | 2433384448 / 7511588864 | 101553 / 204302 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 77.7359245308 | 6437847040 | 4237209600 | 2473484288 / 7511588864 | 99729 / 203990 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 76.1733333333 | 6437847040 | 4248121344 | 2448490496 / 7511588864 | 99934 / 198671 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 78.5433333333 | 6437847040 | 3789504512 | 3116396544 / 9530753024 | 101377 / 201128 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 74.5320911318 | 6437847040 | 4239282176 | 2467590144 / 7511588864 | 101214 / 199589 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 16:21:18,884 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 16:22:45,440 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162993944 | 265428057 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 161568304 | 259598486 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 443168120 | 453888537 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 16:22:45,440 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #30 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 16:22:55,505 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 16:22:55,506 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 16:22:55,506 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:765] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 16:23:06,262 | test | INFO | pool-32-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.68800020218 seconds to complete 2021-05-17 16:23:06,266 | test | INFO | pool-32-thread-6 | [common_lib:sleep:22] Sleep 8.68800020218 seconds. Reason: Wait after rebalance complete 2021-05-17 16:25:15,017 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0589997768402 seconds 2021-05-17 16:25:15,019 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 16:25:15,019 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 16:25:23,562 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 16:35:33,382 | test | INFO | pool-32-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******......*....*****..............*****....*..........*....***X | 59.699 | | 57.300 | .......................................X | 57.300 | | 57.000 | .......................................*****..*........****X | 71.393 | | 57.098 | .......................................******............*****..*...............*****.........*..****X | 114.259 | | 112.710 | ..............................................................................*.....****X | 120.537 | | 94.805 | .................................................................******...******X | 99.107 | | 97.000 | ...................................................................*****...........*......****X | 121.109 | | 111.404 | .............................................................................X | 111.404 | | 106.394 | .........................................................................****X | 106.394 | | 105.500 | .........................................................................*...***X | 110.206 | | 106.506 | ..........................................................................*....*****.X | 114.497 | | 84.197 | ..........................................................*****.........*............****X | 114.418 | | 112.406 | ..............................................................................******.....****X | 120.291 | | 119.391 | ...................................................................................*............*****X | 137.000 | | 120.592 | ...................................................................................*****................*****X | 141.992 | | 121.005 | ....................................................................................****X | 121.005 | | 104.603 | ........................................................................X | 104.603 | | 103.397 | .......................................................................*****..X | 105.700 | | 100.600 | .....................................................................*****................*****X | 122.703 | | 107.614 | ..........................................................................*****X | 107.614 | | 105.604 | .........................................................................X | 105.604 | | 92.395 | ................................................................*****..*.....****X | 103.602 | | 50.201 | ..................................X | 50.201 | | 22.502 | ..............****X | 22.502 | | 1.500 | X | 1.500 | | 1.500 | X | 1.500 | | 0.000 | ****.*.*****X | 3.725 | | 3.500 | .***********X | 3.700 | | 1.897 | ******...*X | 6.287 | | 6.287 | ...***.*X | 7.287 | | 7.287 | ....**X | 7.287 | | 5.587 | ..X | 5.587 | | 0.200 | ******.****X | 1.497 | | 0.697 | X | 0.697 | | 0.000 | *****.*.****X | 3.994 | | 3.093 | .X | 3.093 | | 2.297 | ****X | 2.297 | | 2.096 | X | 2.096 | | 2.096 | ***********..*..*X | 7.589 | | 7.589 | ....**X | 7.589 | | 6.289 | ...X | 6.289 | | 4.695 | ..*****X | 4.695 | | 2.596 | *****...*.****X | 7.991 | | 5.391 | ..X | 5.391 | | 2.897 | .**********X | 4.195 | | 3.995 | .*X | 3.995 | | 3.596 | .***X | 3.596 | | 2.596 | *X | 2.596 | | 2.098 | ****.******.*..******.***X | 9.292 | | 7.991 | ....X | 7.991 | | 6.194 | ...***********.*.**X | 8.892 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:35:33,496 | test | INFO | pool-32-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******............*****........*.........*****....*...................****X | 81.298 | | 62.300 | .......................................X | 62.300 | | 54.300 | ..................................*****........*..............****X | 88.392 | | 86.198 | ......................................................*****X | 86.198 | | 86.193 | ......................................................X | 86.193 | | 85.193 | ......................................................*****......*...*****...............*.......*****...*****X | 138.243 | | 116.407 | ..........................................................................*****X | 116.407 | | 88.900 | ........................................................*****.........*..*****..******...*****X | 113.995 | | 107.105 | ....................................................................*****X | 107.105 | | 97.895 | ..............................................................*****X | 97.895 | | 93.596 | ...........................................................*****X | 93.596 | | 75.691 | ...............................................X | 75.691 | | 61.009 | ......................................*****....*...****X | 71.498 | | 69.597 | ...........................................X | 69.597 | | 66.495 | .........................................***X | 66.495 | | 64.195 | ........................................X | 64.195 | | 57.100 | ...................................****X | 57.100 | | 57.000 | ...................................X | 57.000 | | 51.896 | ................................*****.*****X | 53.796 | | 33.202 | ....................*****.....................*........*****X | 77.700 | | 76.800 | ................................................*****..*..*****.................*****X | 108.613 | | 90.396 | .........................................................******..........******........*..................*****......X | 154.668 | | 152.360 | .................................................................................................****X | 152.360 | | 151.152 | ................................................................................................X | 151.152 | | 150.750 | ................................................................................................****X | 150.750 | | 11.500 | ......*****X | 11.500 | | 4.999 | ..****X | 4.999 | | 4.999 | ..X | 4.999 | | 2.896 | ******..*X | 5.289 | | 5.289 | ..**X | 5.289 | | 5.289 | ..*.............*****..........*........................****X | 78.419 | | 73.319 | ..............................................X | 73.319 | | 60.437 | ......................................*****...*.........******....****X | 85.227 | | 81.021 | ...................................................X | 81.021 | | 75.529 | ...............................................****X | 75.529 | | 73.833 | ..............................................*****X | 73.833 | | 41.769 | ..........................****X | 41.769 | | 31.369 | ...................X | 31.369 | | 4.394 | .******.****X | 4.694 | | 4.094 | .X | 4.094 | | 2.896 | *****.*.*.***X | 6.693 | | 5.393 | ..X | 5.393 | | 2.797 | *******.**********X | 3.896 | | 2.996 | X | 2.996 | | 2.397 | *****.******...****X | 8.292 | | 7.491 | ...X | 7.491 | | 7.191 | ...X | 7.191 | | 6.794 | ...***X | 6.794 | | 5.994 | ..X | 5.994 | | 5.194 | ..******.******.**X | 8.193 | +---------+------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:35:34,151 | test | INFO | pool-32-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******....................................................................................................*****X | 1614.260 | | 482.990 | ............................*****X | 482.990 | | 398.905 | .......................X | 398.905 | | 355.592 | .....................****X | 355.592 | | 227.799 | .............******.........................**X | 637.732 | | 637.732 | ......................................**X | 637.732 | | 397.089 | .......................**X | 397.089 | | 397.089 | .......................**.******.*....****X | 490.076 | | 464.111 | ...........................X | 464.111 | | 361.175 | .....................****X | 361.175 | | 327.706 | ...................X | 327.706 | | 235.571 | .............*****.....******.....****X | 389.596 | | 273.318 | ...............X | 273.318 | | 237.097 | .............*****.*****X | 250.019 | | 236.188 | .............******.******..******.****X | 304.486 | | 216.113 | ............X | 216.113 | | 191.329 | ..........*****.******.******.*****.X | 243.501 | | 238.382 | .............**X | 238.382 | | 238.382 | .............**.....*.*****.*.****X | 363.818 | | 251.709 | ..............X | 251.709 | | 224.694 | ............****X | 224.694 | | 209.684 | ...........X | 209.684 | | 208.201 | ...........*****....*.****.*.....*.***X | 403.550 | | 359.544 | .....................X | 359.544 | | 217.287 | ............******..****X | 245.005 | | 243.505 | ..............*.*****.**.*****..*****..*......*X | 463.396 | | 463.396 | ...........................***....******..*.....****X | 636.074 | | 591.012 | ...................................X | 591.012 | | 584.811 | ...................................****X | 584.811 | | 451.925 | ..........................X | 451.925 | | 399.182 | .......................****X | 399.182 | | 321.062 | ..................X | 321.062 | | 293.693 | .................****X | 293.693 | | 281.400 | ................******..*****X | 314.767 | | 272.674 | ...............******..*****X | 303.934 | | 292.704 | .................*****..*......*****..*.........****X | 600.318 | | 571.660 | ..................................X | 571.660 | | 545.500 | ................................****X | 545.500 | | 495.573 | .............................X | 495.573 | | 461.578 | ...........................****X | 461.578 | | 380.278 | ......................X | 380.278 | | 331.681 | ...................*****.....**.*X | 429.196 | | 429.196 | .........................**.........**..***X | 611.215 | | 602.030 | ....................................**.***X | 616.194 | | 523.711 | ...............................X | 523.711 | | 500.768 | ..............................****X | 500.768 | | 500.154 | .............................X | 500.154 | | 476.823 | ............................*****..*..***X | 543.946 | | 525.044 | ...............................X | 525.044 | | 493.958 | .............................*****......**.***X | 610.498 | | 591.070 | ...................................*X | 591.070 | | 575.120 | ..................................*****..***X | 597.706 | +---------+------------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 16:35:34,395 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 16:35:34,447 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 16:35:45,275 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 16:35:45,545 | test | INFO | pool-32-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.231674389146 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 34.9978332611 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 42.9832907674 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 43.1492808453 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 35.1561718724 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 16:35:45,625 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:35:50,742 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:35:52,983 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 16:35:55,801 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:36:00,865 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:36:05,948 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:36:11,209 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.09 2021-05-17 16:36:16,296 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 2.94 2021-05-17 16:36:21,391 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 7.93 2021-05-17 16:36:26,467 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 15.99 2021-05-17 16:36:31,579 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 21.42 2021-05-17 16:36:32,279 | test | INFO | pool-32-thread-26 | [table_view:display:72] Keys failed in bucket1:bN-21-251000:PhQX4pI-21-435000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1508 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1547 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1536 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1498 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:34,279 | test | INFO | pool-32-thread-20 | [table_view:display:72] Keys failed in bucket1:bN-21-251000:-m-21-435000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1183 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1150 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1172 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1056 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1013 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:36,697 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 23.56 2021-05-17 16:36:41,780 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 26.07 2021-05-17 16:36:46,867 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 28.45 2021-05-17 16:36:49,670 | test | INFO | pool-32-thread-16 | [table_view:display:72] Keys failed in bucket1:_default:GO9bitS_b73BnUgFezXw-38-109000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:50,598 | test | INFO | pool-32-thread-26 | [table_view:display:72] Keys failed in bucket1:bN-21-251000:PhQX4pI-21-435000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1749 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1653 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1786 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1668 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1616 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:52,000 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 31.09 2021-05-17 16:36:52,615 | test | INFO | pool-32-thread-4 | [table_view:display:72] Keys failed in bucket1:LIkW__IAxr1kS-50-234000:EOxqS-50-386000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-22 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:52,617 | test | INFO | pool-32-thread-4 | [table_view:display:72] Keys failed in bucket1:LIkW__IAxr1kS-50-234000:EOxqS-50-386000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-22 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:54,444 | test | INFO | pool-32-thread-20 | [table_view:display:72] Keys failed in bucket1:bN-21-251000:-m-21-435000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1269 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1336 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1351 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1252 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1230 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1387 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:36:57,096 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 39.74 2021-05-17 16:37:02,196 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 43.56 2021-05-17 16:37:07,273 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 49.74 2021-05-17 16:37:12,374 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 52.94 2021-05-17 16:37:14,441 | test | INFO | pool-32-thread-28 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1339 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1342 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:16,572 | test | INFO | pool-32-thread-17 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:0%z6zNPnBA985N6nf-20-743000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1904 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1918 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1808 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1912 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1802 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1882 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1894 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:17,431 | test | INFO | pool-32-thread-10 | [table_view:display:72] Keys failed in bucket2:w1qwPPIM%-38-100000:IxlBjr-38-103000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:17,506 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 55.56 2021-05-17 16:37:20,196 | test | INFO | pool-32-thread-8 | [table_view:display:72] Keys failed in bucket2:FI52amqmtL%8WvoFgwb-19-12000:-UAT7LOt3YzXbiJVm6-19-220000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1622 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1600 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1667 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1645 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1724 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1790 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1761 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1696 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:22,596 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 57.42 2021-05-17 16:37:22,598 | test | INFO | pool-32-thread-14 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-304 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-225 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-248 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-279 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-202 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-233 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-363 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-294 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:25,650 | test | INFO | pool-32-thread-23 | [table_view:display:72] Keys failed in bucket2:FI52amqmtL%8WvoFgwb-19-12000:-UAT7LOt3YzXbiJVm6-19-220000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-925 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-965 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-973 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-951 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-875 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:27,701 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 59.32 2021-05-17 16:37:30,174 | test | INFO | pool-32-thread-28 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1507 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1565 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1444 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1401 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1525 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1591 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1452 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1497 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1542 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:32,418 | test | INFO | pool-32-thread-7 | [table_view:display:72] Keys failed in bucket2:G--hD34S77KGwanCd-49-77000:V7RjQodmFkRA8e-49-378000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1067 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:32,420 | test | INFO | pool-32-thread-7 | [table_view:display:72] Keys failed in bucket2:G--hD34S77KGwanCd-49-77000:V7RjQodmFkRA8e-49-378000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1067 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:32,806 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 69.24 2021-05-17 16:37:36,683 | test | INFO | pool-32-thread-10 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:0%z6zNPnBA985N6nf-20-743000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-119 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-145 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-93 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:37,733 | test | INFO | pool-32-thread-26 | [table_view:display:72] Keys failed in default:_default:pnXwzBj1FKm92i0v%-38-124000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1391 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:37,904 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 80.48 2021-05-17 16:37:42,464 | test | INFO | pool-32-thread-27 | [table_view:display:72] Keys failed in default:mN_rf-38-135000:AdyfUNPN-38-137000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-914 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-973 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:42,989 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 82.75 2021-05-17 16:37:43,805 | test | INFO | pool-32-thread-8 | [table_view:display:72] Keys failed in bucket2:FI52amqmtL%8WvoFgwb-19-12000:-UAT7LOt3YzXbiJVm6-19-220000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1839 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1901 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1891 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:48,117 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 85.38 2021-05-17 16:37:48,657 | test | INFO | pool-32-thread-14 | [table_view:display:72] Keys failed in bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-552 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:37:53,230 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 86.83 2021-05-17 16:37:58,322 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: running, progress: 89.26 2021-05-17 16:38:00,211 | test | INFO | pool-32-thread-1 | [table_view:display:72] Keys failed in default:Bb-%Sua%Wh7yrZ-51-411000:4QDY70rvtEzcB-51-591000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1045 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:38:00,213 | test | INFO | pool-32-thread-1 | [table_view:display:72] Keys failed in default:Bb-%Sua%Wh7yrZ-51-411000:4QDY70rvtEzcB-51-591000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1045 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:38:01,204 | test | INFO | pool-32-thread-30 | [table_view:display:72] Keys failed in default:mN_rf-38-135000:oXzWmby8qVjfQ1ZO7T-38-135000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1016 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1177 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:38:01,740 | test | INFO | pool-32-thread-8 | [table_view:display:72] Keys failed in default:GgyiF-38-132000:shPjl4V%bu_JAa4OPG-38-133000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1016 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1124 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 16:38:08,428 | test | INFO | pool-32-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 16:38:08,473 | test | INFO | pool-32-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 142.930000067 sec 2021-05-17 16:40:13,151 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 14 docs from 1000 to 1960 2021-05-17 16:40:13,151 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 6 docs from 0 to 1000 2021-05-17 16:40:53,782 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 16:40:53,782 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 11 docs from 1000 to 1960 2021-05-17 16:40:53,783 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 13 docs from 1000 to 1960 2021-05-17 16:41:08,234 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 15 docs from 1000 to 1960 2021-05-17 16:41:08,236 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 16:41:08,236 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 16 docs from 1000 to 1960 2021-05-17 16:41:08,236 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 13 docs from 0 to 1000 2021-05-17 16:41:08,237 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 16:41:26,342 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 16:41:26,342 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 16:41:26,342 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 1 docs from 980 to 1440 2021-05-17 16:41:26,344 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 16:41:26,344 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 6 docs from 0 to 1000 2021-05-17 16:41:35,743 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 4 docs from 980 to 1440 2021-05-17 16:41:35,743 | test | ERROR | pool-32-thread-13 | [task:call:3809] Failed to load 4 docs from 980 to 1440 2021-05-17 16:48:13,256 | test | INFO | pool-32-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................*..........................................................................***X | 269.918 | | 141.192 | ...................................................X | 141.192 | | 87.182 | ...............................****.......****.........*....****.....***X | 154.861 | | 152.251 | .......................................................X | 152.251 | | 115.149 | .........................................****...*.......***X | 141.072 | | 107.775 | ......................................X | 107.775 | | 20.493 | ......****............................................................****X | 181.619 | | 162.207 | ...........................................................***.................*....***X | 221.272 | | 85.287 | ..............................*****....................*.......****....................****X | 213.037 | | 159.161 | .........................................................X | 159.161 | | 152.468 | .......................................................****X | 152.468 | | 140.784 | ...................................................****X | 140.784 | | 124.758 | .............................................*...****......*..*****..................*......***X | 219.754 | | 213.158 | .............................................................................X | 213.158 | | 200.872 | .........................................................................****X | 200.872 | | 165.057 | ............................................................****X | 165.057 | | 154.355 | ........................................................X | 154.355 | | 143.470 | ....................................................*****..****X | 150.797 | | 146.297 | .....................................................X | 146.297 | | 139.094 | ..................................................****X | 139.094 | | 125.183 | .............................................*.....*****...........*****..*X | 175.278 | | 168.177 | .............................................................***X | 168.177 | | 130.080 | ...............................................X | 130.080 | | 126.691 | .............................................*****..........*..*****.****X | 160.574 | | 159.279 | ..........................................................*****X | 160.578 | | 122.791 | ............................................X | 122.791 | | 111.796 | ........................................***X | 111.796 | | 108.199 | .......................................X | 108.199 | | 98.879 | ...................................****X | 98.879 | | 74.104 | ..........................X | 74.104 | | 66.319 | .......................****X | 66.319 | | 46.365 | ................*****X | 46.365 | | 6.425 | .*****......*.....................****X | 80.228 | | 75.635 | ...........................X | 75.635 | | 74.335 | ..........................****X | 74.335 | | 25.371 | ........****X | 25.371 | | 19.974 | ......X | 19.974 | | 1.600 | ****X | 1.600 | | 1.200 | *.*****....*****........*............*****.*....***X | 81.130 | | 71.937 | .........................X | 71.937 | | 38.855 | .............****X | 38.855 | | 1.600 | *****X | 1.600 | | 0.400 | **********X | 2.001 | | 1.101 | X | 1.101 | | 0.600 | *****.*.***X | 5.403 | | 4.602 | X | 4.602 | | 2.802 | ****X | 2.802 | | 2.402 | X | 2.402 | | 1.601 | *********X | 2.602 | | 2.201 | X | 2.201 | | 1.801 | *****........................*****X | 66.526 | | 36.505 | ............****X | 36.505 | | 30.105 | ..........X | 30.105 | | 1.501 | *****...*****....*................****X | 62.816 | | 14.403 | ....*****X | 14.403 | | 0.600 | ***********.**********X | 5.204 | | 2.400 | X | 2.400 | | 2.400 | ****X | 2.400 | | 0.600 | ******.*****.....*.......................****X | 80.417 | | 53.504 | ..................****X | 53.504 | | 47.409 | ................X | 47.409 | | 28.504 | .........****X | 28.504 | | 1.401 | ***********.*.*****X | 7.202 | | 6.601 | .**X | 6.601 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:48:13,967 | test | INFO | pool-32-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...........*...........****................****X | 87.300 | | 55.377 | .......................****.......................................***X | 144.127 | | 126.127 | ......................................................X | 126.127 | | 67.385 | ............................****.****X | 70.164 | | 27.606 | ...........***X | 27.606 | | 24.307 | .........*..........****.........................****X | 104.656 | | 101.100 | ...........................................********...*.****..................*...........................***X | 212.732 | | 194.236 | ....................................................................................X | 194.236 | | 126.620 | ......................................................*****................*....****...................****X | 216.172 | | 200.432 | .......................................................................................X | 200.432 | | 199.437 | ......................................................................................****X | 199.437 | | 141.984 | .............................................................X | 141.984 | | 130.094 | ........................................................****....*...............****X | 174.864 | | 131.181 | ........................................................*****.*.*..****.......X | 154.758 | | 146.570 | ...............................................................*****....****X | 156.395 | | 142.991 | .............................................................X | 142.991 | | 134.491 | ..........................................................*****..........*****...*....*****...................*.....****X | 227.564 | | 127.693 | .......................................................****X | 127.693 | | 124.684 | .....................................................*..****X | 128.782 | | 119.781 | ...................................................*****X | 119.781 | | 94.687 | ........................................X | 94.687 | | 93.387 | ........................................***X | 93.387 | | 88.585 | .....................................*.****X | 88.796 | | 57.792 | ........................X | 57.792 | | 47.695 | ...................*****.....****X | 58.716 | | 43.272 | ..................X | 43.272 | | 39.170 | ................****X | 39.170 | | 18.996 | .......X | 18.996 | | 2.989 | ****...*...*****...*.............*****.*.****X | 58.131 | | 55.068 | .......................*****..........*.................****X | 117.498 | | 110.194 | ...............................................X | 110.194 | | 97.626 | .........................................****X | 97.626 | | 59.353 | .........................****X | 59.353 | | 45.547 | ...................X | 45.547 | | 21.682 | ........***X | 21.682 | | 16.983 | ......X | 16.983 | | 1.000 | ****X | 1.000 | | 0.000 | ****************X | 0.600 | | 0.200 | ******..****X | 6.005 | | 1.399 | X | 1.399 | | 0.200 | ****X | 0.200 | | 0.200 | ************.****X | 2.502 | | 1.000 | ****X | 1.000 | | 0.400 | *****X | 0.400 | | 0.400 | *****........*.................................****X | 93.822 | | 70.518 | .............................******....*X | 78.418 | | 78.418 | .................................***....*****X | 88.402 | | 39.899 | ................X | 39.899 | | 27.913 | ...........****X | 27.913 | | 15.223 | .....X | 15.223 | | 4.802 | .***X | 4.802 | | 3.802 | X | 3.802 | | 2.401 | *****......******.............*.......***X | 63.714 | | 26.129 | ..........X | 26.129 | | 2.001 | *****.******........................****X | 58.402 | | 46.110 | ...................X | 46.110 | | 40.009 | ................****X | 40.009 | | 0.500 | *****.............*.....................****X | 78.209 | | 37.729 | ...............X | 37.729 | | 6.100 | .****X | 6.100 | | 2.800 | ***X | 2.800 | +---------+---------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:48:14,538 | test | INFO | pool-32-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *......................................*............***X | 367.866 | | 211.080 | ............................X | 211.080 | | 61.990 | .......****.....................*........**X | 269.811 | | 93.553 | ...........*****.................****..*......*X | 272.184 | | 272.184 | ....................................*X | 272.184 | | 166.672 | ......................X | 166.672 | | 166.672 | ......................*X | 166.672 | | 166.672 | ......................*X | 166.672 | | 85.249 | ..........X | 85.249 | | 59.673 | .......***..*....****........*...**X | 183.808 | | 150.191 | ...................X | 150.191 | | 140.206 | ..................****..*.***X | 160.269 | | 86.963 | ...........*****.........................*...****...............*................*.......**X | 565.097 | | 520.588 | .......................................................................X | 520.588 | | 337.758 | .............................................*****...............*....***X | 475.748 | | 358.607 | ................................................X | 358.607 | | 182.690 | ........................*****..X | 195.171 | | 192.762 | .........................**X | 192.762 | | 192.762 | .........................*...*.................****X | 336.650 | | 314.628 | ..........................................******...........****X | 390.510 | | 331.982 | .............................................X | 331.982 | | 294.694 | .......................................****X | 294.694 | | 252.083 | .................................X | 252.083 | | 232.984 | ...............................***X | 232.984 | | 207.563 | ...........................X | 207.563 | | 155.180 | ....................*****.....*****X | 193.482 | | 114.781 | ..............*****...........*............*.....***X | 312.981 | | 267.658 | ....................................X | 267.658 | | 252.166 | .................................*****...................****X | 387.838 | | 295.151 | .......................................X | 295.151 | | 275.161 | .....................................**********..............*....**X | 404.148 | | 404.148 | .......................................................*X | 404.148 | | 245.667 | .................................*****......*..*..****........................*.................****X | 613.636 | | 452.230 | .............................................................X | 452.230 | | 291.916 | .......................................****X | 291.916 | | 185.613 | ........................*****......*..................*****.......*...****.*....****X | 468.770 | | 430.955 | ..........................................................X | 430.955 | | 245.725 | .................................****X | 245.725 | | 172.164 | ......................*****..*.....*****...............*...*****.........*.......***X | 462.976 | | 455.714 | ..............................................................X | 455.714 | | 427.935 | ..........................................................****X | 427.935 | | 346.233 | ..............................................******..................****X | 470.497 | | 425.697 | ..........................................................X | 425.697 | | 391.550 | .....................................................*****.X | 401.370 | | 400.252 | ......................................................****.*.........****X | 476.024 | | 410.276 | .......................................................**X | 410.276 | | 410.276 | .......................................................**....*........***X | 496.617 | | 471.317 | ................................................................X | 471.317 | | 325.134 | ............................................***X | 325.134 | | 325.134 | ............................................*........******.........****X | 449.636 | | 269.160 | ....................................****X | 269.160 | | 244.169 | ................................X | 244.169 | | 188.758 | .........................****X | 188.758 | | 184.955 | ........................*...*****.....******................*****...........*...****X | 457.679 | | 434.273 | ...........................................................****X | 434.273 | | 405.773 | .......................................................X | 405.773 | | 399.779 | ......................................................****X | 399.779 | | 371.896 | ..................................................X | 371.896 | | 277.369 | .....................................*****......................................*****X | 549.298 | | 278.352 | .....................................****X | 278.352 | | 194.961 | ..........................X | 194.961 | | 144.705 | ...................*****...........******.............*.............***X | 414.576 | | 387.676 | ....................................................X | 387.676 | | 277.332 | .....................................*****.................................******.............................***X | 721.470 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:48:14,584 | test | WARNING | Thread-108061 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:0%z6zNPnBA985N6nf-20-743000 update failed keys from task: [u'test_collections-1926', u'test_collections-1904', u'test_collections-1808', u'test_collections-1941', u'test_collections-1888', u'test_collections-1802', u'test_collections-1847', u'test_collections-1882', u'test_collections-1918', u'test_collections-1876', u'test_collections-1912', u'test_collections-1957', u'test_collections-1894'] 2021-05-17 16:48:14,584 | test | WARNING | Thread-108062 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 update failed keys from task: [u'test_collections-1270', u'test_collections-1507', u'test_collections-1565', u'test_collections-1444', u'test_collections-1401', u'test_collections-1525', u'test_collections-1360', u'test_collections-1560', u'test_collections-1342', u'test_collections-1542', u'test_collections-1520', u'test_collections-1339', u'test_collections-1539', u'test_collections-1591', u'test_collections-1452', u'test_collections-1497'] 2021-05-17 16:48:14,592 | test | WARNING | Thread-108096 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:FI52amqmtL%8WvoFgwb-19-12000:-UAT7LOt3YzXbiJVm6-19-220000 update failed keys from task: [u'test_collections-1706', u'test_collections-1622', u'test_collections-1600', u'test_collections-1667', u'test_collections-1645', u'test_collections-1724', u'test_collections-1901', u'test_collections-1627', u'test_collections-1761', u'test_collections-1958', u'test_collections-1839', u'test_collections-1790', u'test_collections-1891', u'test_collections-1696'] 2021-05-17 16:48:14,595 | test | WARNING | Thread-108102 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:G--hD34S77KGwanCd-49-77000:V7RjQodmFkRA8e-49-378000 create failed keys from task: [u'test_collections-1059', u'test_collections-1067'] 2021-05-17 16:48:14,641 | test | WARNING | Thread-108274 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bN-21-251000:-m-21-435000 update failed keys from task: [u'test_collections-1269', u'test_collections-1183', u'test_collections-1382', u'test_collections-1342', u'test_collections-1387', u'test_collections-1336', u'test_collections-1217', u'test_collections-1150', u'test_collections-1172', u'test_collections-1351', u'test_collections-1252', u'test_collections-1230', u'test_collections-1056', u'test_collections-1013', u'test_collections-1212'] 2021-05-17 16:48:14,641 | test | WARNING | Thread-108278 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:LIkW__IAxr1kS-50-234000:EOxqS-50-386000 create failed keys from task: [u'test_collections-22'] 2021-05-17 16:48:14,641 | test | WARNING | Thread-108273 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bN-21-251000:PhQX4pI-21-435000 update failed keys from task: [u'test_collections-1749', u'test_collections-1508', u'test_collections-1786', u'test_collections-1547', u'test_collections-1668', u'test_collections-1653', u'test_collections-1536', u'test_collections-1616', u'test_collections-1737', u'test_collections-1772', u'test_collections-1498'] 2021-05-17 16:48:14,686 | test | WARNING | Thread-108319 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:pnXwzBj1FKm92i0v%-38-124000 update failed keys from task: [u'test_collections-1391'] 2021-05-17 16:48:14,686 | test | WARNING | Thread-108227 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:GO9bitS_b73BnUgFezXw-38-109000 update failed keys from task: [u'test_collections-6'] 2021-05-17 16:48:14,698 | test | WARNING | Thread-108371 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:GgyiF-38-132000:shPjl4V%bu_JAa4OPG-38-133000 update failed keys from task: [u'test_collections-1016', u'test_collections-1028', u'test_collections-1031', u'test_collections-1124'] 2021-05-17 16:48:14,707 | test | WARNING | Thread-108407 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:mN_rf-38-135000:AdyfUNPN-38-137000 delete failed keys from task: [u'test_collections-914', u'test_collections-973'] 2021-05-17 16:48:14,709 | test | WARNING | Thread-108415 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:mN_rf-38-135000:oXzWmby8qVjfQ1ZO7T-38-135000 update failed keys from task: [u'test_collections-982', u'test_collections-1016', u'test_collections-1177', u'test_collections-1124'] 2021-05-17 16:48:14,710 | test | WARNING | Thread-108428 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Bb-%Sua%Wh7yrZ-51-411000:4QDY70rvtEzcB-51-591000 create failed keys from task: [u'test_collections-1045'] 2021-05-17 16:48:14,753 | test | WARNING | Thread-108178 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:w1qwPPIM%-38-100000:IxlBjr-38-103000 delete failed keys from task: [u'test_collections-3'] 2021-05-17 16:48:15,542 | test | WARNING | Thread-108096 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:FI52amqmtL%8WvoFgwb-19-12000:-UAT7LOt3YzXbiJVm6-19-220000 delete failed keys from task: [u'test_collections-869', u'test_collections-925', u'test_collections-965', u'test_collections-973', u'test_collections-951', u'test_collections-875'] 2021-05-17 16:48:15,582 | test | WARNING | Thread-108062 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:3H9Ak-20-744000 delete failed keys from task: [u'test_collections-304', u'test_collections-225', u'test_collections-248', u'test_collections-202', u'test_collections-341', u'test_collections-363', u'test_collections-460', u'test_collections-439', u'test_collections-279', u'test_collections-233', u'test_collections-552', u'test_collections-294', u'test_collections-392'] 2021-05-17 16:48:15,585 | test | WARNING | Thread-108061 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BPboiGrJ4zdSvJ6P9q-20-628000:0%z6zNPnBA985N6nf-20-743000 delete failed keys from task: [u'test_collections-119', u'test_collections-85', u'test_collections-116', u'test_collections-145', u'test_collections-93', u'test_collections-153'] 2021-05-17 16:48:29,618 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 16:49:45,104 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 16:49:45,105 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 16:50:06,994 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 16:50:06,996 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 16:50:23,437 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 16:50:23,438 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 16:51:10,857 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1320002079 seconds to complete 2021-05-17 16:51:10,871 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 3117.056s OK During the test, Remote Connections: 205, Disconnections: 205 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 30 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_30 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_31 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_forced_hard_failover_rebalance_out,nodes_init=4,nodes_failover=2,forced_hard_failover=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=initial_load,data_load_stage=during,quota_percent=80,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_31', 'nodes_init': '4', 'forced_hard_failover': 'True', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 31, 'data_load_spec': 'initial_load'} test_data_load_collections_with_forced_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 16:51:20,259 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #31 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 16:51:20,944 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #31 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 16:51:21,388 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x11e3584 type=client open_count=1 channel=[id: 0x39a1730d, 0.0.0.0/0.0.0.0:43423] timeout=300.0>) 2021-05-17 16:51:42,249 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x11e3585 type=client open_count=1 channel=[id: 0x88e093fd, 0.0.0.0/0.0.0.0:37608] timeout=300.0>) 2021-05-17 16:51:44,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x11e3586 type=client open_count=1 channel=[id: 0x11814121, 0.0.0.0/0.0.0.0:43101] timeout=300.0>) 2021-05-17 16:51:46,283 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11e3587 type=client open_count=1 channel=[id: 0xe219930a, 0.0.0.0/0.0.0.0:48170] timeout=300.0>) 2021-05-17 16:52:10,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11e3588 type=client open_count=1 channel=[id: 0xfa577da6, 0.0.0.0/0.0.0.0:55229] timeout=300.0>) 2021-05-17 16:52:12,609 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11e3589 type=client open_count=1 channel=[id: 0x6c9b0512, 0.0.0.0/0.0.0.0:38339] timeout=300.0>) 2021-05-17 16:52:14,628 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x11e358a type=client open_count=1 channel=[id: 0xbc63e027, 0.0.0.0/0.0.0.0:51422] timeout=300.0>) 2021-05-17 16:52:38,742 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x11e358b type=client open_count=1 channel=[id: 0xe90ba84b, 0.0.0.0/0.0.0.0:56756] timeout=300.0>) 2021-05-17 16:52:40,759 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x11e358c type=client open_count=1 channel=[id: 0xf5dfcdbd, 0.0.0.0/0.0.0.0:37275] timeout=300.0>) 2021-05-17 16:52:42,778 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x11e358d type=client open_count=1 channel=[id: 0x11593225, 0.0.0.0/0.0.0.0:39817] timeout=300.0>) 2021-05-17 16:53:07,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x11e358e type=client open_count=1 channel=[id: 0x4abb1c16, 0.0.0.0/0.0.0.0:49686] timeout=300.0>) 2021-05-17 16:53:09,127 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x11e358f type=client open_count=1 channel=[id: 0xc7f28df3, 0.0.0.0/0.0.0.0:43088] timeout=300.0>) 2021-05-17 16:53:11,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x11e3590 type=client open_count=1 channel=[id: 0x352bd305, 0.0.0.0/0.0.0.0:43458] timeout=300.0>) 2021-05-17 16:53:35,266 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x11e3591 type=client open_count=1 channel=[id: 0x134a3d6f, 0.0.0.0/0.0.0.0:48755] timeout=300.0>) 2021-05-17 16:53:37,285 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x11e3592 type=client open_count=1 channel=[id: 0x52cc8ec5, 0.0.0.0/0.0.0.0:58753] timeout=300.0>) 2021-05-17 16:53:39,302 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 16:53:56,746 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 16:54:10,542 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 16:54:10,569 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #31 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 16:54:10,609 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #31 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 16:54:21,634 | test | INFO | pool-33-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.76825386244 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 16:54:21,658 | test | INFO | pool-33-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 16:54:31,736 | test | INFO | pool-33-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 16:54:31,762 | test | INFO | pool-33-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.128000021 sec 2021-05-17 16:54:32,036 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 16:55:20,538 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 16:55:20,885 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 852638720 | 111558831 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 298202032 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 1198070608 | 61141213 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 16:55:20,885 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 16:56:02,059 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 16:56:02,075 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 16:56:22,842 | test | INFO | pool-33-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......................**..............................................................................X | 26573.435 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 16:56:22,907 | test | INFO | pool-33-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................**................................................................X | 121.407 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:56:22,976 | test | INFO | pool-33-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.........................**...........................................................................X | 177.107 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 16:56:23,253 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.9735886431 | 6437847040 | 3950002176 | 2816528384 / 7511588864 | 83214 / 232315 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.949763049 | 6437847040 | 4030418944 | 2714050560 / 7511588864 | 81519 / 172396 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 84.2782238243 | 6437847040 | 3616931840 | 3326722048 / 9530753024 | 82977 / 239579 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 81.6707649365 | 6437847040 | 4138422272 | 2635448320 / 7511588864 | 81792 / 140128 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 16:56:36,861 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 16:57:47,720 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 211191392 | 307824032 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 315028912 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 850951616 | 611744346 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 16:57:47,720 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #31 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 16:57:55,933 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during forced_hard_failover_rebalance_out 2021-05-17 16:57:55,934 | test | INFO | MainThread | [collections_rebalance:forced_failover_operation:372] Updating all the bucket replicas to 1 2021-05-17 16:57:56,206 | test | INFO | MainThread | [collections_rebalance:forced_failover_operation:375] failing over nodes [ip:172.23.107.129 port:8091 ssh_username:Administrator, ip:172.23.107.131 port:8091 ssh_username:Administrator] 2021-05-17 16:58:08,496 | test | INFO | pool-33-thread-16 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.44799995422 seconds to complete 2021-05-17 16:58:08,502 | test | INFO | pool-33-thread-16 | [common_lib:sleep:22] Sleep 8.44799995422 seconds. Reason: Wait after rebalance complete 2021-05-17 17:00:17,009 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 17:00:29,128 | test | INFO | pool-33-thread-27 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.19499993324 seconds to complete 2021-05-17 17:00:29,134 | test | INFO | pool-33-thread-27 | [common_lib:sleep:22] Sleep 8.19499993324 seconds. Reason: Wait after rebalance complete 2021-05-17 17:02:37,391 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 17:02:37,444 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 17:02:37,446 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:02:37,575 | test | INFO | pool-33-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 9.06439570695 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 0.31498950035 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 9.11333333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.368314917587 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 17:02:37,601 | test | INFO | pool-33-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:02:38,786 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:02:57,364 | test | INFO | pool-33-thread-26 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................*X | 17773.128 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:02:57,680 | test | INFO | pool-33-thread-6 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **...................................................................................................*X | 41.932 | +-------+---------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 17:02:59,780 | test | INFO | pool-33-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......**.............................................................................................X | 280.531 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:03:01,631 | test | INFO | pool-33-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:03:01,799 | test | INFO | pool-33-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 24.2230000496 sec 2021-05-17 17:03:28,815 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:03:30,289 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:04:03,759 | test | INFO | pool-33-thread-28 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...................................***..............................................................**...*X | 32674.785 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:04:03,816 | test | INFO | pool-33-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....**...............................................*................................................*X | 485.203 | | 52.495 | .........*X | 52.495 | +--------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:04:04,898 | test | INFO | pool-33-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..............***......................................................................................**X | 330.658 | | 152.983 | .............................................*X | 152.983 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:04:07,039 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 17:04:07,040 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 17:04:35,243 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 17:04:35,243 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 17:05:07,484 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 17:05:07,484 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 17:06:09,734 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1749999523 seconds to complete 2021-05-17 17:06:09,750 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 898.722s OK During the test, Remote Connections: 108, Disconnections: 108 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 31 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_31 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_32 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_forced_hard_failover_rebalance_out,nodes_init=5,nodes_failover=3,forced_hard_failover=True,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=initial_load,data_load_stage=during,quota_percent=80,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '3', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_32', 'nodes_init': '5', 'forced_hard_failover': 'True', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 32, 'data_load_spec': 'initial_load'} test_data_load_collections_with_forced_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 17:06:19,141 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #32 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 17:06:19,861 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #32 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 17:06:20,286 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x12293c3 type=client open_count=1 channel=[id: 0xa2b27a49, 0.0.0.0/0.0.0.0:33154] timeout=300.0>) 2021-05-17 17:06:41,109 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x12293c4 type=client open_count=1 channel=[id: 0x332cb4cd, 0.0.0.0/0.0.0.0:46010] timeout=300.0>) 2021-05-17 17:06:43,128 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x12293c5 type=client open_count=1 channel=[id: 0x9d425922, 0.0.0.0/0.0.0.0:34374] timeout=300.0>) 2021-05-17 17:06:45,145 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x12293c6 type=client open_count=1 channel=[id: 0xdffc9838, 0.0.0.0/0.0.0.0:38662] timeout=300.0>) 2021-05-17 17:07:09,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x12293c7 type=client open_count=1 channel=[id: 0x4c3fb3ba, 0.0.0.0/0.0.0.0:50609] timeout=300.0>) 2021-05-17 17:07:11,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x12293c8 type=client open_count=1 channel=[id: 0xb710f845, 0.0.0.0/0.0.0.0:50078] timeout=300.0>) 2021-05-17 17:07:13,533 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x12293c9 type=client open_count=1 channel=[id: 0x606e5243, 0.0.0.0/0.0.0.0:60749] timeout=300.0>) 2021-05-17 17:07:38,038 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x12293ca type=client open_count=1 channel=[id: 0xd7d05b67, 0.0.0.0/0.0.0.0:36343] timeout=300.0>) 2021-05-17 17:07:40,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x12293cb type=client open_count=1 channel=[id: 0x8862ddef, 0.0.0.0/0.0.0.0:46915] timeout=300.0>) 2021-05-17 17:07:42,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x12293cc type=client open_count=1 channel=[id: 0x908e3616, 0.0.0.0/0.0.0.0:38084] timeout=300.0>) 2021-05-17 17:07:44,089 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x12293cd type=client open_count=1 channel=[id: 0x81ca2f40, 0.0.0.0/0.0.0.0:47298] timeout=300.0>) 2021-05-17 17:08:08,091 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x12293ce type=client open_count=1 channel=[id: 0xe3530a32, 0.0.0.0/0.0.0.0:38193] timeout=300.0>) 2021-05-17 17:08:10,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x12293cf type=client open_count=1 channel=[id: 0xce97bc87, 0.0.0.0/0.0.0.0:51478] timeout=300.0>) 2021-05-17 17:08:12,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x12293d0 type=client open_count=1 channel=[id: 0xb3945eeb, 0.0.0.0/0.0.0.0:44170] timeout=300.0>) 2021-05-17 17:08:36,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x12293d1 type=client open_count=1 channel=[id: 0xbe70a35e, 0.0.0.0/0.0.0.0:52381] timeout=300.0>) 2021-05-17 17:08:38,220 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x12293d2 type=client open_count=1 channel=[id: 0xaa4189ff, 0.0.0.0/0.0.0.0:38226] timeout=300.0>) 2021-05-17 17:08:40,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 17:08:57,434 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 17:09:11,203 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 17:09:11,224 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #32 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 17:09:11,260 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #32 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 17:09:25,290 | test | INFO | pool-34-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.1394286857 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 17:09:25,315 | test | INFO | pool-34-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:09:35,391 | test | INFO | pool-34-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:09:35,418 | test | INFO | pool-34-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1270000935 sec 2021-05-17 17:09:35,717 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 17:10:28,099 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 17:10:28,430 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 799304568 | 137425525 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 288066128 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1119231096 | 91070220 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 17:10:28,430 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 17:11:17,677 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:11:17,690 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:11:39,384 | test | INFO | pool-34-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 158.977 | | 132.847 | ..................................................................................X | 132.847 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:11:39,540 | test | INFO | pool-34-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..........................................................................**.........................X | 24313.400 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:11:41,240 | test | INFO | pool-34-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **................................................................................................**....*X | 132.952 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:11:41,519 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9198758075 | 6437847040 | 4195561472 | 2604662784 / 7511588864 | 89386 / 270286 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 100 | 6437847040 | 4165443584 | 2585300992 / 7511588864 | 88653 / 214250 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.1413595141 | 6437847040 | 4209283072 | 2604519424 / 7511588864 | 87185 / 228008 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.8416666667 | 6437847040 | 3688845312 | 3252375552 / 9530753024 | 89177 / 226600 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7883420124 | 6437847040 | 4171431936 | 2593886208 / 7511588864 | 89434 / 186238 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 17:11:55,388 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:13:24,013 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211792808 | 308381692 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320196160 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 837009544 | 626577844 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 17:13:24,013 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #32 test_data_load_collections_with_forced_hard_failover_rebalance_out ========= 2021-05-17 17:13:34,305 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during forced_hard_failover_rebalance_out 2021-05-17 17:13:34,306 | test | INFO | MainThread | [collections_rebalance:forced_failover_operation:372] Updating all the bucket replicas to 1 2021-05-17 17:13:34,578 | test | INFO | MainThread | [collections_rebalance:forced_failover_operation:375] failing over nodes [ip:172.23.107.129 port:8091 ssh_username:Administrator, ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 17:13:46,756 | test | INFO | pool-34-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.42300009727 seconds to complete 2021-05-17 17:13:46,759 | test | INFO | pool-34-thread-21 | [common_lib:sleep:22] Sleep 8.42300009727 seconds. Reason: Wait after rebalance complete 2021-05-17 17:15:55,263 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0759999752045 seconds 2021-05-17 17:16:07,365 | test | INFO | pool-34-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.11500000954 seconds to complete 2021-05-17 17:16:07,374 | test | INFO | pool-34-thread-6 | [common_lib:sleep:22] Sleep 8.11500000954 seconds. Reason: Wait after rebalance complete 2021-05-17 17:18:15,555 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0629999637604 seconds 2021-05-17 17:18:26,697 | test | INFO | pool-34-thread-28 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.28299999237 seconds to complete 2021-05-17 17:18:26,703 | test | INFO | pool-34-thread-28 | [common_lib:sleep:22] Sleep 8.28299999237 seconds. Reason: Wait after rebalance complete 2021-05-17 17:20:35,046 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 3 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 17:20:35,099 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 17:20:35,099 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:20:35,236 | test | INFO | pool-34-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.391666666667 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 8.62152297462 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.45333333333 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 7.47345789096 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.546648445052 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 17:20:35,265 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:20:36,408 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:20:44,467 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.36 2021-05-17 17:20:49,801 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 2.96 2021-05-17 17:20:54,941 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 4.8 2021-05-17 17:20:55,259 | test | INFO | pool-34-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................................................................*..................................X | 137.519 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:20:56,456 | test | INFO | pool-34-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 22334.093 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:20:56,763 | test | INFO | pool-34-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...........................................................*........................................*X | 171.635 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:21:00,032 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 10.64 2021-05-17 17:21:05,144 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 15.65 2021-05-17 17:21:10,249 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 22.34 2021-05-17 17:21:14,335 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:21:15,364 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 26.44 2021-05-17 17:21:15,684 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:21:20,598 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 27.51 2021-05-17 17:21:25,772 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 28.33 2021-05-17 17:21:30,881 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 29.14 2021-05-17 17:21:36,219 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 29.89 2021-05-17 17:21:41,437 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 30.6 2021-05-17 17:21:46,687 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 31.45 2021-05-17 17:21:51,953 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 32.53 2021-05-17 17:21:52,592 | test | INFO | pool-34-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.*......*.....**........................................................................................X | 292.145 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:21:53,592 | test | INFO | pool-34-thread-10 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...............................................................................**....................**.X | 30065.888 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:21:55,302 | test | INFO | pool-34-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+-------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+-------+ | 0.000 | *******.................................*...................................................................*X | 9.003 | +-------+----------------------------------------------------------------------------------------------------------------+-------+ 2021-05-17 17:21:57,052 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 35.29 2021-05-17 17:22:02,141 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 42.87 2021-05-17 17:22:07,220 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 49.98 2021-05-17 17:22:12,301 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 61.57 2021-05-17 17:22:17,404 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 71.13 2021-05-17 17:22:22,467 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 86.81 2021-05-17 17:22:27,540 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 89.39 2021-05-17 17:22:32,609 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 91.73 2021-05-17 17:22:37,680 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 94.17 2021-05-17 17:22:42,746 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 96.58 2021-05-17 17:22:47,826 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: running, progress: 98.83 2021-05-17 17:22:57,907 | test | INFO | pool-34-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:22:58,108 | test | INFO | pool-34-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 142.874000072 sec 2021-05-17 17:22:59,075 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 17:22:59,075 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 17:23:16,729 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 17:23:16,730 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 17:23:40,710 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 17:23:40,711 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 17:24:39,789 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1330001354 seconds to complete 2021-05-17 17:24:42,069 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1112.300s OK During the test, Remote Connections: 124, Disconnections: 124 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 32 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_32 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 17h 6m 50s 2 actionable tasks: 2 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.7 -P client_version=default -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.7 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/ssh.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/eagles_all_around.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main scripts/install.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_rebalance_crud_on_collections.conf -m rest > Task :compileJava UP-TO-DATE > Task :testrunner Filename: conf/collections/collections_rebalance_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_rebalance_crud_on_collections'} Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_1', 'nodes_init': '3', 'rebalance_moves_per_node': '32', 'GROUP': 'rebalance_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 1, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 17:25:05,861 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:25:06,869 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:25:08,903 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0x30fa7e00, 0.0.0.0/0.0.0.0:57845] timeout=300.0>) 2021-05-17 17:25:30,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xbec0c5ab, 0.0.0.0/0.0.0.0:39458] timeout=300.0>) 2021-05-17 17:25:32,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x5493066b, 0.0.0.0/0.0.0.0:35038] timeout=300.0>) 2021-05-17 17:25:34,747 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0x472472e9, 0.0.0.0/0.0.0.0:50999] timeout=300.0>) 2021-05-17 17:25:59,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xe1a6133d, 0.0.0.0/0.0.0.0:42722] timeout=300.0>) 2021-05-17 17:26:01,418 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xc1af7cf4, 0.0.0.0/0.0.0.0:54068] timeout=300.0>) 2021-05-17 17:26:03,457 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x167dccc1, 0.0.0.0/0.0.0.0:41860] timeout=300.0>) 2021-05-17 17:26:27,890 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0x30189c5a, 0.0.0.0/0.0.0.0:42792] timeout=300.0>) 2021-05-17 17:26:29,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0x79e443e7, 0.0.0.0/0.0.0.0:36329] timeout=300.0>) 2021-05-17 17:26:31,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x8970352e, 0.0.0.0/0.0.0.0:55317] timeout=300.0>) 2021-05-17 17:26:56,263 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x91ef67ad, 0.0.0.0/0.0.0.0:35517] timeout=300.0>) 2021-05-17 17:26:58,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x1f85da3a, 0.0.0.0/0.0.0.0:43403] timeout=300.0>) 2021-05-17 17:27:00,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x7bcf0f27, 0.0.0.0/0.0.0.0:55153] timeout=300.0>) 2021-05-17 17:27:02,352 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x79348125, 0.0.0.0/0.0.0.0:51269] timeout=300.0>) 2021-05-17 17:27:26,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0xfa0a6620, 0.0.0.0/0.0.0.0:59585] timeout=300.0>) 2021-05-17 17:27:28,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x98fa9070, 0.0.0.0/0.0.0.0:32981] timeout=300.0>) 2021-05-17 17:27:30,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 17:27:47,578 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 17:28:01,359 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 17:28:01,384 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:28:01,427 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:28:08,267 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.5186092638 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 17:28:08,296 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:28:18,401 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:28:18,430 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1640000343 sec 2021-05-17 17:28:18,756 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 17:29:00,828 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 17:29:01,236 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 629145600 | 582262744 | 67173602 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 629145600 | 225519136 | 102 | | default | couchbase | 3 | none | 0 | 0 | 6291456000 | 929109160 | 39122184 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 17:29:01,239 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 17:29:32,631 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:29:32,660 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:29:56,467 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..***..................................................................................................*X | 21721.482 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:29:58,454 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..........................***..........................................................................*X | 136.249 | | 127.915 | ............................................................................................X | 127.915 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:30:00,461 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........................***..........................................................................**..*X | 134.610 | +-------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:30:00,891 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 100 | 6437847040 | 4047388672 | 2689314816 / 7511588864 | 169000 / 209751 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 100 | 6437847040 | 4084056064 | 2716323840 / 7511588864 | 167888 / 242680 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 100 | 6437847040 | 3588894720 | 3298459648 / 9530753024 | 168400 / 218851 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 17:30:12,690 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:31:07,352 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 164421912 | 234415540 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 629145600 | 238006640 | 102 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 626705944 | 473235398 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 17:31:07,355 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:31:13,779 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 17:31:15,479 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 17:31:15,480 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before rebalance_in 2021-05-17 17:31:15,483 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:31:23,058 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:31:23,342 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-17 17:31:35,279 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 22.3451327434 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 27.9518475171 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 26.8166666667 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 17:31:35,737 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:31:40,959 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:31:46,427 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:31:48,398 | test | INFO | pool-2-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................................................................................*..................X | 25242.923 | | 10154.623 | .......................................X | 10154.623 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:31:48,894 | test | INFO | pool-2-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 4360.201 | | 1848.690 | .........................................X | 1848.690 | | 41.207 | X | 41.207 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:31:49,019 | test | INFO | pool-2-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **................................*....................................................................X | 5552.323 | | 1271.588 | .....................X | 1271.588 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:31:51,686 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 1.73 2021-05-17 17:31:56,877 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.26 2021-05-17 17:32:02,006 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 8.97 2021-05-17 17:32:07,114 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.71 2021-05-17 17:32:12,799 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 15.97 2021-05-17 17:32:17,913 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 18.23 2021-05-17 17:32:23,039 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 22.09 2021-05-17 17:32:28,236 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 25.62 2021-05-17 17:32:33,423 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 30.7 2021-05-17 17:32:38,503 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 17:32:43,980 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 35.59 2021-05-17 17:32:49,121 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 41.84 2021-05-17 17:32:54,273 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 46.38 2021-05-17 17:32:59,690 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 51.65 2021-05-17 17:33:04,799 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 56.94 2021-05-17 17:33:10,421 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.34 2021-05-17 17:33:15,507 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 17:33:20,642 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.08 2021-05-17 17:33:25,811 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 71.61 2021-05-17 17:33:30,907 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 76.55 2021-05-17 17:33:36,088 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.84 2021-05-17 17:33:41,184 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 82.36 2021-05-17 17:33:46,285 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 86.78 2021-05-17 17:33:51,404 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 88.25 2021-05-17 17:33:56,490 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 91.47 2021-05-17 17:34:01,608 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 94.24 2021-05-17 17:34:07,255 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 97.09 2021-05-17 17:34:17,358 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:34:17,400 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 162.121999979 sec 2021-05-17 17:34:30,250 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:35:53,496 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 17:35:55,595 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 17:35:56,608 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 17:35:56,608 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 17:36:15,401 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 56% 2021-05-17 17:36:15,404 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 75% 2021-05-17 17:36:35,191 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 17:36:35,194 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 17:37:39,604 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.2160000801 seconds to complete 2021-05-17 17:37:39,627 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 763.843s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_1 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_2', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 17:37:49,493 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:37:50,338 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:37:50,993 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1bb type=client open_count=1 channel=[id: 0xecd045c9, 0.0.0.0/0.0.0.0:34021] timeout=300.0>) 2021-05-17 17:38:12,683 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1bc type=client open_count=1 channel=[id: 0xc2c14559, 0.0.0.0/0.0.0.0:56748] timeout=300.0>) 2021-05-17 17:38:14,711 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1bd type=client open_count=1 channel=[id: 0x0fd10673, 0.0.0.0/0.0.0.0:47245] timeout=300.0>) 2021-05-17 17:38:16,737 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d1be type=client open_count=1 channel=[id: 0x61066880, 0.0.0.0/0.0.0.0:41023] timeout=300.0>) 2021-05-17 17:38:40,573 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d1bf type=client open_count=1 channel=[id: 0x6ac46048, 0.0.0.0/0.0.0.0:43576] timeout=300.0>) 2021-05-17 17:38:42,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d1c0 type=client open_count=1 channel=[id: 0xed094f64, 0.0.0.0/0.0.0.0:50846] timeout=300.0>) 2021-05-17 17:38:44,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1c1 type=client open_count=1 channel=[id: 0x4fd05de3, 0.0.0.0/0.0.0.0:53940] timeout=300.0>) 2021-05-17 17:39:08,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1c2 type=client open_count=1 channel=[id: 0x3c0f25ae, 0.0.0.0/0.0.0.0:60559] timeout=300.0>) 2021-05-17 17:39:10,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1c3 type=client open_count=1 channel=[id: 0x6066b45e, 0.0.0.0/0.0.0.0:59201] timeout=300.0>) 2021-05-17 17:39:12,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1c4 type=client open_count=1 channel=[id: 0xbfa6bb18, 0.0.0.0/0.0.0.0:39640] timeout=300.0>) 2021-05-17 17:39:37,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1c5 type=client open_count=1 channel=[id: 0x85ca33b9, 0.0.0.0/0.0.0.0:47555] timeout=300.0>) 2021-05-17 17:39:39,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1c6 type=client open_count=1 channel=[id: 0x6c9fd7d5, 0.0.0.0/0.0.0.0:47824] timeout=300.0>) 2021-05-17 17:39:41,171 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1c7 type=client open_count=1 channel=[id: 0xbb15a885, 0.0.0.0/0.0.0.0:41867] timeout=300.0>) 2021-05-17 17:40:05,619 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1c8 type=client open_count=1 channel=[id: 0x3ee60dd7, 0.0.0.0/0.0.0.0:37972] timeout=300.0>) 2021-05-17 17:40:07,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1c9 type=client open_count=1 channel=[id: 0xac5ce37f, 0.0.0.0/0.0.0.0:33870] timeout=300.0>) 2021-05-17 17:40:09,676 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 17:40:29,275 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 17:40:43,730 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 17:40:43,755 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:40:43,796 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #2 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:40:50,967 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.30145159409 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 17:40:50,993 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:41:01,068 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:41:01,095 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.129999876 sec 2021-05-17 17:41:01,361 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 17:41:42,805 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 17:41:43,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 629145600 | 635145592 | 80991975 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 629145600 | 215366688 | 102 | | default | couchbase | 3 | none | 0 | 0 | 6291456000 | 947070616 | 62494459 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 17:41:43,232 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 17:42:12,217 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:42:12,232 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:42:33,181 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..................***..................................................................................*X | 26966.702 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:42:34,217 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......................................****.............................................................*X | 153.015 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:42:35,559 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.............................................***.......................................................**X | 186.019 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:42:35,931 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 100 | 6437847040 | 3964305408 | 2778529792 / 7511588864 | 109982 / 147469 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8683399163 | 6437847040 | 4042641408 | 2736984064 / 7511588864 | 168800 / 301768 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9732923817 | 6437847040 | 3520561152 | 3391979520 / 9530753024 | 107782 / 141392 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 17:42:47,867 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:43:41,392 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 159302536 | 233329666 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 629145600 | 236821344 | 102 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 601422648 | 465911220 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 17:43:41,394 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_rebalance_in ========= 2021-05-17 17:43:47,769 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in 2021-05-17 17:43:47,769 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-17 17:43:47,828 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:43:55,058 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:43:57,838 | test | INFO | pool-4-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 13.3585559759 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 18.5930234496 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 13.8533333333 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 17:43:58,224 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:44:03,559 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:44:09,759 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.12 2021-05-17 17:44:15,555 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.76 2021-05-17 17:44:18,733 | test | INFO | pool-4-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 8731.149 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:44:19,049 | test | INFO | pool-4-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.................................*...................................................................*X | 2677.163 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:44:19,170 | test | INFO | pool-4-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 1268.323 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:44:20,884 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 3.48 2021-05-17 17:44:26,069 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 6.05 2021-05-17 17:44:31,194 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 8.75 2021-05-17 17:44:36,351 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 11.03 2021-05-17 17:44:41,505 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 13.77 2021-05-17 17:44:46,607 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 16.48 2021-05-17 17:44:51,726 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 19.3 2021-05-17 17:44:56,829 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 22.04 2021-05-17 17:45:02,033 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 25.39 2021-05-17 17:45:07,142 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 28.72 2021-05-17 17:45:12,263 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 31.99 2021-05-17 17:45:18,440 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 34.28 2021-05-17 17:45:23,556 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 41.83 2021-05-17 17:45:28,694 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 47.16 2021-05-17 17:45:33,842 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 52.58 2021-05-17 17:45:38,977 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 57.49 2021-05-17 17:45:44,104 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 61.87 2021-05-17 17:45:49,367 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.52 2021-05-17 17:45:54,484 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 68.48 2021-05-17 17:45:59,750 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 73.76 2021-05-17 17:46:04,951 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.04 2021-05-17 17:46:10,095 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 81.74 2021-05-17 17:46:15,246 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 84.5 2021-05-17 17:46:20,348 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 87.08 2021-05-17 17:46:25,464 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 89.86 2021-05-17 17:46:30,579 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 92.76 2021-05-17 17:46:35,684 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 95.48 2021-05-17 17:46:40,799 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: running, progress: 98.29 2021-05-17 17:46:50,960 | test | INFO | pool-4-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:46:51,002 | test | INFO | pool-4-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 173.1650002 sec 2021-05-17 17:47:03,690 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:48:30,105 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 17:48:30,107 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 17:49:05,736 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 96% 2021-05-17 17:49:05,737 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 92% 2021-05-17 17:49:34,025 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 17:49:34,026 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 17:50:36,648 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1670000553 seconds to complete 2021-05-17 17:50:36,664 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 777.061s During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_2 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_3', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'rebalance_with_collection_crud', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 3, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 17:50:47,221 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_rebalance_out ========= 2021-05-17 17:50:48,013 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_rebalance_out ========= 2021-05-17 17:50:48,638 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb9f18 type=client open_count=1 channel=[id: 0x8c596cfa, 0.0.0.0/0.0.0.0:58599] timeout=300.0>) 2021-05-17 17:51:09,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb9f19 type=client open_count=1 channel=[id: 0xbf78b6d0, 0.0.0.0/0.0.0.0:33165] timeout=300.0>) 2021-05-17 17:51:11,732 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xb9f1a type=client open_count=1 channel=[id: 0x3d0c0a83, 0.0.0.0/0.0.0.0:52631] timeout=300.0>) 2021-05-17 17:51:13,753 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb9f1b type=client open_count=1 channel=[id: 0xb25409a4, 0.0.0.0/0.0.0.0:52023] timeout=300.0>) 2021-05-17 17:51:37,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb9f1c type=client open_count=1 channel=[id: 0x6466be8a, 0.0.0.0/0.0.0.0:57508] timeout=300.0>) 2021-05-17 17:51:39,783 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xb9f1d type=client open_count=1 channel=[id: 0xc33fc20f, 0.0.0.0/0.0.0.0:40537] timeout=300.0>) 2021-05-17 17:51:41,805 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb9f1e type=client open_count=1 channel=[id: 0xbee1b1ec, 0.0.0.0/0.0.0.0:56911] timeout=300.0>) 2021-05-17 17:52:05,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb9f1f type=client open_count=1 channel=[id: 0xbd9c2dc5, 0.0.0.0/0.0.0.0:42960] timeout=300.0>) 2021-05-17 17:52:07,739 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0xb9f20 type=client open_count=1 channel=[id: 0xf8b78558, 0.0.0.0/0.0.0.0:53765] timeout=300.0>) 2021-05-17 17:52:09,759 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb9f21 type=client open_count=1 channel=[id: 0x1b981785, 0.0.0.0/0.0.0.0:52073] timeout=300.0>) 2021-05-17 17:52:33,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb9f22 type=client open_count=1 channel=[id: 0xda4ff9c0, 0.0.0.0/0.0.0.0:40104] timeout=300.0>) 2021-05-17 17:52:35,684 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xb9f23 type=client open_count=1 channel=[id: 0x1b724c62, 0.0.0.0/0.0.0.0:48252] timeout=300.0>) 2021-05-17 17:52:37,704 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb9f24 type=client open_count=1 channel=[id: 0x28e0c366, 0.0.0.0/0.0.0.0:53270] timeout=300.0>) 2021-05-17 17:53:01,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb9f25 type=client open_count=1 channel=[id: 0x0504e0f8, 0.0.0.0/0.0.0.0:59948] timeout=300.0>) 2021-05-17 17:53:03,796 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0xb9f26 type=client open_count=1 channel=[id: 0x29611a73, 0.0.0.0/0.0.0.0:60765] timeout=300.0>) 2021-05-17 17:53:05,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 17:53:23,914 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 17:53:39,694 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 17:53:39,717 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_rebalance_out ========= 2021-05-17 17:53:39,763 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #3 test_data_load_collections_with_rebalance_out ========= 2021-05-17 17:53:53,686 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.2549457509 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 17:53:53,710 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:54:03,789 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 17:54:03,818 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1319999695 sec 2021-05-17 17:54:04,075 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 17:54:50,986 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 17:54:51,401 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 908720440 | 106113533 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 294425136 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1136902952 | 55665703 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 17:54:51,403 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 17:55:41,532 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:55:41,543 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:56:04,155 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *......................................................................................*.......*.......X | 144.957 | | 114.553 | ..............................................................................X | 114.553 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:56:05,841 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........................***........................................................................*X | 23847.342 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:56:06,161 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.............................***.......................................................................*X | 172.070 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 17:56:06,487 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.8702055079 | 6437847040 | 4172992512 | 2635382784 / 7511588864 | 62561 / 154937 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.9481683359 | 6437847040 | 4105502720 | 2654846976 / 7511588864 | 100200 / 262990 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.4106704858 | 6437847040 | 4120305664 | 2687221760 / 7511588864 | 61872 / 189639 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.6866718888 | 6437847040 | 3598397440 | 3335299072 / 9530753024 | 101800 / 287064 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7899964999 | 6437847040 | 4185673728 | 2635304960 / 7511588864 | 101400 / 282612 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 17:56:21,707 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 17:57:51,605 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212189912 | 299956617 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 318682768 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 825629480 | 617985854 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 17:57:51,608 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_rebalance_out ========= 2021-05-17 17:58:01,878 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 17:58:03,559 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 17:58:03,559 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before rebalance_out 2021-05-17 17:58:03,561 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 17:58:11,944 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 17:58:12,171 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-17 17:58:12,565 | test | INFO | pool-5-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 96.5634478851 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 59.0406619691 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 50.7419310722 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 62.9789638098 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 48.5949531651 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 17:58:12,594 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:58:17,778 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:58:27,184 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 17:58:28,539 | test | INFO | pool-5-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......................*.............................................................................X | 5281.518 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:58:28,607 | test | INFO | pool-5-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.............................*.......................................................................X | 20427.870 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 17:58:28,786 | test | INFO | pool-5-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *................*....................................................................................X | 5240.136 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 17:58:32,315 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.72 2021-05-17 17:58:37,417 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 6.52 2021-05-17 17:58:42,506 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 9.64 2021-05-17 17:58:47,624 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 12.85 2021-05-17 17:58:52,736 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 16.15 2021-05-17 17:58:57,880 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 18.64 2021-05-17 17:59:02,990 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 20.9 2021-05-17 17:59:08,105 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 25.42 2021-05-17 17:59:13,213 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 28.38 2021-05-17 17:59:18,361 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.7 2021-05-17 17:59:23,460 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 17:59:28,529 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 17:59:33,630 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 17:59:38,789 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.15 2021-05-17 17:59:43,908 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 48.13 2021-05-17 17:59:49,000 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 54.41 2021-05-17 17:59:54,197 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 61.04 2021-05-17 17:59:59,306 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:00:04,377 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:00:09,469 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:00:14,632 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.88 2021-05-17 18:00:19,780 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 75.7 2021-05-17 18:00:24,908 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 79.84 2021-05-17 18:00:30,009 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 85.04 2021-05-17 18:00:35,121 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 90.89 2021-05-17 18:00:40,221 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 93.59 2021-05-17 18:00:45,334 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 98.29 2021-05-17 18:00:50,411 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:00:55,479 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:01:05,578 | test | INFO | pool-5-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:01:05,730 | test | INFO | pool-5-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 173.167000055 sec 2021-05-17 18:01:14,280 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:02:03,900 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 18:02:05,964 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 18:02:06,960 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 18:02:06,960 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 18:02:23,414 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 18:02:23,414 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 18:02:43,210 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 18:02:43,210 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 18:03:38,016 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1340000629 seconds to complete 2021-05-17 18:03:38,032 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 781.186s OK During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_3 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_4', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 18:03:47,500 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_rebalance_out ========= 2021-05-17 18:03:48,237 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_rebalance_out ========= 2021-05-17 18:03:48,684 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1167b2 type=client open_count=1 channel=[id: 0x8e9e5f7f, 0.0.0.0/0.0.0.0:60938] timeout=300.0>) 2021-05-17 18:04:09,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1167b3 type=client open_count=1 channel=[id: 0x4e89b947, 0.0.0.0/0.0.0.0:52195] timeout=300.0>) 2021-05-17 18:04:11,641 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1167b4 type=client open_count=1 channel=[id: 0x8895030c, 0.0.0.0/0.0.0.0:49863] timeout=300.0>) 2021-05-17 18:04:13,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1167b5 type=client open_count=1 channel=[id: 0x1e2ff1cb, 0.0.0.0/0.0.0.0:42889] timeout=300.0>) 2021-05-17 18:04:37,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1167b6 type=client open_count=1 channel=[id: 0x47c4c8c7, 0.0.0.0/0.0.0.0:44486] timeout=300.0>) 2021-05-17 18:04:39,573 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1167b7 type=client open_count=1 channel=[id: 0x91e21de7, 0.0.0.0/0.0.0.0:59966] timeout=300.0>) 2021-05-17 18:04:41,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1167b8 type=client open_count=1 channel=[id: 0x92857f25, 0.0.0.0/0.0.0.0:38906] timeout=300.0>) 2021-05-17 18:05:05,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1167b9 type=client open_count=1 channel=[id: 0x670fd0d2, 0.0.0.0/0.0.0.0:48750] timeout=300.0>) 2021-05-17 18:05:07,530 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1167ba type=client open_count=1 channel=[id: 0x3f1d1748, 0.0.0.0/0.0.0.0:53769] timeout=300.0>) 2021-05-17 18:05:09,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1167bb type=client open_count=1 channel=[id: 0xdde3f468, 0.0.0.0/0.0.0.0:37717] timeout=300.0>) 2021-05-17 18:05:33,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1167bc type=client open_count=1 channel=[id: 0xf2d09a0d, 0.0.0.0/0.0.0.0:58185] timeout=300.0>) 2021-05-17 18:05:35,877 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1167bd type=client open_count=1 channel=[id: 0xabbbacdc, 0.0.0.0/0.0.0.0:42564] timeout=300.0>) 2021-05-17 18:05:37,898 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1167be type=client open_count=1 channel=[id: 0x41c92fb3, 0.0.0.0/0.0.0.0:43304] timeout=300.0>) 2021-05-17 18:06:01,950 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1167bf type=client open_count=1 channel=[id: 0x6134605d, 0.0.0.0/0.0.0.0:47495] timeout=300.0>) 2021-05-17 18:06:03,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1167c0 type=client open_count=1 channel=[id: 0x8e826190, 0.0.0.0/0.0.0.0:49934] timeout=300.0>) 2021-05-17 18:06:05,993 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 18:06:23,319 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 18:06:37,030 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 18:06:37,055 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_rebalance_out ========= 2021-05-17 18:06:37,094 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #4 test_data_load_collections_with_rebalance_out ========= 2021-05-17 18:06:50,236 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.04994916751 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:06:50,260 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:07:00,348 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:07:00,384 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.146999836 sec 2021-05-17 18:07:00,760 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 18:07:46,292 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 18:07:46,634 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 923520016 | 110702429 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 278549088 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1221871848 | 67882151 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 18:07:46,635 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 18:08:32,010 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:08:32,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:08:54,178 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *................................................................***....................................X | 137.287 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:08:54,315 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......................***..............................................................................X | 24714.005 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:08:54,394 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *....................................................................................................**X | 58.776 | | 13.490 | .....................X | 13.490 | +--------+----------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 18:08:54,756 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.6377895192 | 6437847040 | 4185927680 | 2603634688 / 7511588864 | 61859 / 154726 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 98.8532951098 | 6437847040 | 4073369600 | 2672451584 / 7511588864 | 61261 / 187406 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.6876401881 | 6437847040 | 4146311168 | 2654007296 / 7511588864 | 60508 / 151531 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.7679581991 | 6437847040 | 3628560384 | 3307679744 / 9530753024 | 61278 / 161443 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7926412528 | 6437847040 | 4149911552 | 2673721344 / 7511588864 | 60836 / 161810 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 18:09:09,779 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:10:40,963 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 213202312 | 302188347 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320794544 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 824421704 | 615108217 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 18:10:40,964 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_rebalance_out ========= 2021-05-17 18:10:51,198 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_out 2021-05-17 18:10:51,200 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-17 18:10:51,293 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:10:51,505 | test | INFO | pool-6-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 17.3928261884 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 9.48031601053 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 11.4094295285 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 9.14166666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 10.495 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:10:51,555 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:10:56,825 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:11:00,647 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:11:02,351 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.3 2021-05-17 18:11:07,690 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.5 2021-05-17 18:11:13,030 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.68 2021-05-17 18:11:14,885 | test | INFO | pool-6-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................................................................................X | 627.461 | +-------+--------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:11:15,917 | test | INFO | pool-6-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 3616.180 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:11:18,207 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 4.07 2021-05-17 18:11:18,242 | test | INFO | pool-6-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................................................................................................*X | 950.111 | +-------+---------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:11:23,328 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 7.02 2021-05-17 18:11:28,421 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 9.23 2021-05-17 18:11:33,542 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 11.63 2021-05-17 18:11:38,670 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 14.5 2021-05-17 18:11:43,812 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 17.46 2021-05-17 18:11:48,923 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 21.14 2021-05-17 18:11:54,038 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 24.37 2021-05-17 18:11:59,174 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 27.84 2021-05-17 18:12:04,335 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.35 2021-05-17 18:12:09,437 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 32.26 2021-05-17 18:12:14,533 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:12:19,596 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:12:25,664 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.03 2021-05-17 18:12:30,793 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 44.57 2021-05-17 18:12:35,930 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 49.87 2021-05-17 18:12:41,071 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 55.94 2021-05-17 18:12:46,223 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 61.66 2021-05-17 18:12:51,368 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.15 2021-05-17 18:12:56,480 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:13:01,584 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:13:07,188 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 68.84 2021-05-17 18:13:12,410 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 75.35 2021-05-17 18:13:17,576 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 79.15 2021-05-17 18:13:22,664 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 83.0 2021-05-17 18:13:27,769 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 87.71 2021-05-17 18:13:32,910 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 92.99 2021-05-17 18:13:37,996 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 96.83 2021-05-17 18:13:43,099 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 99.37 2021-05-17 18:13:48,206 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:13:53,275 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:14:03,378 | test | INFO | pool-6-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:14:03,585 | test | INFO | pool-6-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 192.082000017 sec 2021-05-17 18:14:12,201 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:15:02,756 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 18:15:02,756 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 18:15:19,003 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 18:15:19,003 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 18:15:42,822 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 18:15:42,823 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 18:16:33,770 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1289999485 seconds to complete 2021-05-17 18:16:33,786 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 775.708s OK During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_4 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_5', 'nodes_init': '4', 'rebalance_moves_per_node': '32', 'GROUP': 'rebalance_with_collection_crud', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 5, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 18:16:43,257 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:16:43,960 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:16:44,621 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x173383 type=client open_count=1 channel=[id: 0x0f845df9, 0.0.0.0/0.0.0.0:56048] timeout=300.0>) 2021-05-17 18:17:05,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x173384 type=client open_count=1 channel=[id: 0x5c04ba65, 0.0.0.0/0.0.0.0:58889] timeout=300.0>) 2021-05-17 18:17:07,434 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x173385 type=client open_count=1 channel=[id: 0x5f209b86, 0.0.0.0/0.0.0.0:34721] timeout=300.0>) 2021-05-17 18:17:09,450 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x173386 type=client open_count=1 channel=[id: 0x8d424d62, 0.0.0.0/0.0.0.0:40974] timeout=300.0>) 2021-05-17 18:17:33,467 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x173387 type=client open_count=1 channel=[id: 0x29f7b15d, 0.0.0.0/0.0.0.0:42647] timeout=300.0>) 2021-05-17 18:17:35,486 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x173388 type=client open_count=1 channel=[id: 0xb619f3f4, 0.0.0.0/0.0.0.0:43109] timeout=300.0>) 2021-05-17 18:17:37,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x173389 type=client open_count=1 channel=[id: 0x88da629f, 0.0.0.0/0.0.0.0:47289] timeout=300.0>) 2021-05-17 18:18:01,650 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x17338a type=client open_count=1 channel=[id: 0x43a5fa4d, 0.0.0.0/0.0.0.0:58432] timeout=300.0>) 2021-05-17 18:18:03,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x17338b type=client open_count=1 channel=[id: 0x89ef226c, 0.0.0.0/0.0.0.0:33874] timeout=300.0>) 2021-05-17 18:18:05,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x17338c type=client open_count=1 channel=[id: 0x30f4cbfe, 0.0.0.0/0.0.0.0:47959] timeout=300.0>) 2021-05-17 18:18:30,112 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x17338d type=client open_count=1 channel=[id: 0xdeb994cc, 0.0.0.0/0.0.0.0:52552] timeout=300.0>) 2021-05-17 18:18:32,131 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x17338e type=client open_count=1 channel=[id: 0x1d35cd0a, 0.0.0.0/0.0.0.0:42303] timeout=300.0>) 2021-05-17 18:18:34,151 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x17338f type=client open_count=1 channel=[id: 0x8985e50b, 0.0.0.0/0.0.0.0:38628] timeout=300.0>) 2021-05-17 18:18:58,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x173390 type=client open_count=1 channel=[id: 0x9edf8fac, 0.0.0.0/0.0.0.0:59355] timeout=300.0>) 2021-05-17 18:19:00,630 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x173391 type=client open_count=1 channel=[id: 0xa85d5b2d, 0.0.0.0/0.0.0.0:36049] timeout=300.0>) 2021-05-17 18:19:02,653 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 18:19:19,898 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 18:19:33,674 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 18:19:33,698 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:19:33,743 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #5 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:19:44,017 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.85990233496 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:19:44,042 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:19:54,115 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:19:54,141 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.123000145 sec 2021-05-17 18:19:54,427 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 18:20:44,947 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 18:20:45,309 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 806617232 | 116943382 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 282421776 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 1065211200 | 102649965 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 18:20:45,311 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 18:21:26,704 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:21:26,729 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:21:49,094 | test | INFO | pool-7-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***............................................................................................*........X | 134.592 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:21:49,151 | test | INFO | pool-7-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................*.....................................................................*...........X | 22719.142 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:21:50,068 | test | INFO | pool-7-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...............*......*...............................................................................X | 217.130 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:21:50,354 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 100 | 6437847040 | 4010729472 | 2736640000 / 7511588864 | 109944 / 292648 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9479262905 | 6437847040 | 4061011968 | 2718613504 / 7511588864 | 115679 / 334759 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9745767021 | 6437847040 | 3571261440 | 3338625024 / 9530753024 | 113206 / 325189 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9726943819 | 6437847040 | 4133994496 | 2679476224 / 7511588864 | 112511 / 320382 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 18:22:02,895 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:23:15,461 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 216823872 | 306703154 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 316114576 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 842682880 | 603317585 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 18:23:15,463 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:23:23,740 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 18:23:25,364 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 18:23:25,365 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before swap_rebalance 2021-05-17 18:23:25,365 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:23:33,740 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:23:33,943 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-17 18:23:49,345 | test | INFO | pool-7-thread-14 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 97.7177117493 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 97.1244190323 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 99.6966435072 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 98.8124299004 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:23:49,381 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:23:49,789 | test | INFO | pool-7-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 6046.141 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:23:51,342 | test | INFO | pool-7-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.........................................................*...........................................X | 26818.263 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:23:52,865 | test | INFO | pool-7-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......................................*.............................................................X | 2634.257 | | 2154.046 | ................................................................................X | 2154.046 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:23:54,539 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:23:59,647 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 2.55 2021-05-17 18:24:04,825 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 5.62 2021-05-17 18:24:09,953 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 9.51 2021-05-17 18:24:15,055 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 13.36 2021-05-17 18:24:20,164 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 15.45 2021-05-17 18:24:25,328 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 17.63 2021-05-17 18:24:30,477 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 21.19 2021-05-17 18:24:35,617 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 24.47 2021-05-17 18:24:40,779 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 28.28 2021-05-17 18:24:45,858 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 32.69 2021-05-17 18:24:51,026 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:24:56,091 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:25:01,236 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.84 2021-05-17 18:25:06,599 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 39.41 2021-05-17 18:25:11,704 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 46.01 2021-05-17 18:25:16,776 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 50.46 2021-05-17 18:25:22,414 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 56.04 2021-05-17 18:25:27,746 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 62.07 2021-05-17 18:25:32,812 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.02 2021-05-17 18:25:37,874 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:25:42,937 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:25:48,026 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.81 2021-05-17 18:25:53,276 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 71.3 2021-05-17 18:25:58,351 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 72.61 2021-05-17 18:26:03,431 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 76.4 2021-05-17 18:26:08,674 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 78.69 2021-05-17 18:26:13,809 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 81.62 2021-05-17 18:26:18,917 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 84.44 2021-05-17 18:26:24,003 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 87.7 2021-05-17 18:26:29,092 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 90.89 2021-05-17 18:26:34,750 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 93.55 2021-05-17 18:26:39,819 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 97.72 2021-05-17 18:26:44,888 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:26:49,950 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:26:56,055 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:27:06,233 | test | INFO | pool-7-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:27:06,405 | test | INFO | pool-7-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 197.06099987 sec 2021-05-17 18:27:14,621 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:28:04,134 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 18:28:06,119 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 18:28:07,078 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 18:28:07,078 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 18:28:20,053 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 18:28:20,055 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 18:28:43,927 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 18:28:43,927 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 18:29:39,457 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1569998264 seconds to complete 2021-05-17 18:29:39,473 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 785.713s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_5 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_6', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 6, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 18:29:49,880 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #6 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:29:50,703 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #6 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:29:51,369 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1cfe84 type=client open_count=1 channel=[id: 0xfb9e971d, 0.0.0.0/0.0.0.0:47318] timeout=300.0>) 2021-05-17 18:30:12,301 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1cfe85 type=client open_count=1 channel=[id: 0x3b29b764, 0.0.0.0/0.0.0.0:59080] timeout=300.0>) 2021-05-17 18:30:14,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1cfe86 type=client open_count=1 channel=[id: 0x0b21eaa1, 0.0.0.0/0.0.0.0:54528] timeout=300.0>) 2021-05-17 18:30:16,338 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1cfe87 type=client open_count=1 channel=[id: 0x87aa829b, 0.0.0.0/0.0.0.0:37212] timeout=300.0>) 2021-05-17 18:30:40,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1cfe88 type=client open_count=1 channel=[id: 0x0547690a, 0.0.0.0/0.0.0.0:60833] timeout=300.0>) 2021-05-17 18:30:42,273 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1cfe89 type=client open_count=1 channel=[id: 0x56fe2097, 0.0.0.0/0.0.0.0:53003] timeout=300.0>) 2021-05-17 18:30:44,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1cfe8a type=client open_count=1 channel=[id: 0xc839140f, 0.0.0.0/0.0.0.0:43795] timeout=300.0>) 2021-05-17 18:31:08,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1cfe8b type=client open_count=1 channel=[id: 0x5b9e4431, 0.0.0.0/0.0.0.0:51557] timeout=300.0>) 2021-05-17 18:31:10,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1cfe8c type=client open_count=1 channel=[id: 0xa51eedc8, 0.0.0.0/0.0.0.0:35728] timeout=300.0>) 2021-05-17 18:31:12,242 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1cfe8d type=client open_count=1 channel=[id: 0x5606c405, 0.0.0.0/0.0.0.0:50657] timeout=300.0>) 2021-05-17 18:31:36,256 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1cfe8e type=client open_count=1 channel=[id: 0x99113caf, 0.0.0.0/0.0.0.0:52128] timeout=300.0>) 2021-05-17 18:31:38,276 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1cfe8f type=client open_count=1 channel=[id: 0x4e00a690, 0.0.0.0/0.0.0.0:42651] timeout=300.0>) 2021-05-17 18:31:40,295 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1cfe90 type=client open_count=1 channel=[id: 0xf1c3484b, 0.0.0.0/0.0.0.0:48431] timeout=300.0>) 2021-05-17 18:32:04,571 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1cfe91 type=client open_count=1 channel=[id: 0xf3abe599, 0.0.0.0/0.0.0.0:58620] timeout=300.0>) 2021-05-17 18:32:06,589 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1cfe92 type=client open_count=1 channel=[id: 0xdc01708d, 0.0.0.0/0.0.0.0:49794] timeout=300.0>) 2021-05-17 18:32:08,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 18:32:25,716 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 18:32:39,418 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 18:32:39,441 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #6 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:32:39,480 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #6 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:32:49,971 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.87008116802 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:32:49,996 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:33:00,082 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:33:00,118 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1470000744 sec 2021-05-17 18:33:00,469 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 18:33:51,926 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 18:33:52,226 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 767173376 | 113155942 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 286372144 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 1086372272 | 83197687 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 18:33:52,227 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 18:34:29,009 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:34:29,020 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:34:49,976 | test | INFO | pool-8-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *....................*................................................................................X | 146.930 | | 137.801 | ............................................................................................X | 137.801 | +---------+---------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:34:50,243 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................X | 211.399 | | 68.816 | ...............................X | 68.816 | +--------+---------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:34:51,881 | test | INFO | pool-8-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....................................................*.......................*........................X | 26742.944 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:34:52,247 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 100 | 6437847040 | 4088274944 | 2644033536 / 7511588864 | 105407 / 290261 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8425987944 | 6437847040 | 4107202560 | 2670649344 / 7511588864 | 102997 / 233223 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 100 | 6437847040 | 3575402496 | 3327070208 / 9530753024 | 104477 / 302469 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 100 | 6437847040 | 4124868608 | 2670448640 / 7511588864 | 104091 / 287543 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 18:35:04,775 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:36:17,289 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 210809696 | 298849862 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 315045168 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 882560000 | 609898757 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 18:36:17,289 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #6 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 18:36:25,565 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during swap_rebalance 2021-05-17 18:36:25,565 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-17 18:36:29,145 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:36:29,312 | test | INFO | pool-8-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 18.3327222426 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 17.3713771437 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 18.9316666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 17.7110762975 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:36:29,378 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:36:34,519 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:36:38,108 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:36:39,900 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:36:45,150 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:36:50,316 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.11 2021-05-17 18:36:55,648 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.18 2021-05-17 18:37:00,790 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.48 2021-05-17 18:37:05,901 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.97 2021-05-17 18:37:06,085 | test | INFO | pool-8-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******....................................................................................................X | 2636.434 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:37:07,782 | test | INFO | pool-8-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***......................***..............................................................................X | 3152.424 | | 2627.006 | ..................................................................................X | 2627.006 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:37:08,986 | test | INFO | pool-8-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..............***...................................................*...................................X | 21854.992 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:37:11,005 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 2.71 2021-05-17 18:37:16,207 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 5.04 2021-05-17 18:37:21,321 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 7.17 2021-05-17 18:37:26,411 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 9.49 2021-05-17 18:37:31,506 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 12.29 2021-05-17 18:37:36,654 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 15.47 2021-05-17 18:37:41,762 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 18.62 2021-05-17 18:37:46,881 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 21.91 2021-05-17 18:37:52,013 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 24.7 2021-05-17 18:37:57,128 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 28.1 2021-05-17 18:38:02,221 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.46 2021-05-17 18:38:07,359 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:38:12,424 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:38:17,522 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:38:22,686 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 37.32 2021-05-17 18:38:27,792 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-17 18:38:32,954 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 47.02 2021-05-17 18:38:38,040 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 51.96 2021-05-17 18:38:43,168 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 56.82 2021-05-17 18:38:48,276 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 61.2 2021-05-17 18:38:53,434 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.59 2021-05-17 18:38:58,523 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:39:03,598 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:39:08,815 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 67.49 2021-05-17 18:39:13,908 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.46 2021-05-17 18:39:19,032 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 73.13 2021-05-17 18:39:24,140 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 75.83 2021-05-17 18:39:29,239 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 78.54 2021-05-17 18:39:34,338 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 81.29 2021-05-17 18:39:39,405 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 84.08 2021-05-17 18:39:44,512 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 86.67 2021-05-17 18:39:49,614 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 89.39 2021-05-17 18:39:54,703 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 92.19 2021-05-17 18:39:59,812 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 94.96 2021-05-17 18:40:04,914 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 97.54 2021-05-17 18:40:09,983 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:40:15,045 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:40:20,230 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:40:30,365 | test | INFO | pool-8-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:40:30,515 | test | INFO | pool-8-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 241.203999996 sec 2021-05-17 18:40:38,733 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:41:28,832 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 18:41:28,832 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 18:41:44,753 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 91% 2021-05-17 18:41:44,753 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 84% 2021-05-17 18:41:58,914 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 18:41:58,915 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 18:42:57,167 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1370000839 seconds to complete 2021-05-17 18:42:57,183 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 797.626s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_6 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_7 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_7', 'nodes_init': '4', 'rebalance_moves_per_node': '32', 'GROUP': 'rebalance_with_collection_crud', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 7, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 18:43:06,562 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #7 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:43:07,236 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #7 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:43:07,792 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x22d1e1 type=client open_count=1 channel=[id: 0xd5a7df51, 0.0.0.0/0.0.0.0:57959] timeout=300.0>) 2021-05-17 18:43:28,602 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x22d1e2 type=client open_count=1 channel=[id: 0xfbf9f494, 0.0.0.0/0.0.0.0:46165] timeout=300.0>) 2021-05-17 18:43:30,624 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x22d1e3 type=client open_count=1 channel=[id: 0x70520b56, 0.0.0.0/0.0.0.0:60041] timeout=300.0>) 2021-05-17 18:43:32,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x22d1e4 type=client open_count=1 channel=[id: 0xf6e27761, 0.0.0.0/0.0.0.0:33974] timeout=300.0>) 2021-05-17 18:43:56,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x22d1e5 type=client open_count=1 channel=[id: 0xc90c29e0, 0.0.0.0/0.0.0.0:45583] timeout=300.0>) 2021-05-17 18:43:58,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x22d1e6 type=client open_count=1 channel=[id: 0x3ba54bcb, 0.0.0.0/0.0.0.0:42070] timeout=300.0>) 2021-05-17 18:44:00,706 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x22d1e7 type=client open_count=1 channel=[id: 0x26803a1e, 0.0.0.0/0.0.0.0:48959] timeout=300.0>) 2021-05-17 18:44:24,637 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x22d1e8 type=client open_count=1 channel=[id: 0x2d412c0d, 0.0.0.0/0.0.0.0:50260] timeout=300.0>) 2021-05-17 18:44:26,654 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x22d1e9 type=client open_count=1 channel=[id: 0xf6b6df59, 0.0.0.0/0.0.0.0:57077] timeout=300.0>) 2021-05-17 18:44:28,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x22d1ea type=client open_count=1 channel=[id: 0x3df3beee, 0.0.0.0/0.0.0.0:49422] timeout=300.0>) 2021-05-17 18:44:52,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x22d1eb type=client open_count=1 channel=[id: 0x9c630588, 0.0.0.0/0.0.0.0:56347] timeout=300.0>) 2021-05-17 18:44:54,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x22d1ec type=client open_count=1 channel=[id: 0x90ad4eb6, 0.0.0.0/0.0.0.0:52041] timeout=300.0>) 2021-05-17 18:44:56,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x22d1ed type=client open_count=1 channel=[id: 0xd2484076, 0.0.0.0/0.0.0.0:38832] timeout=300.0>) 2021-05-17 18:45:21,066 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x22d1ee type=client open_count=1 channel=[id: 0xaa3735fe, 0.0.0.0/0.0.0.0:36225] timeout=300.0>) 2021-05-17 18:45:23,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x22d1ef type=client open_count=1 channel=[id: 0xe166b9d5, 0.0.0.0/0.0.0.0:46271] timeout=300.0>) 2021-05-17 18:45:25,102 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 18:45:42,177 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 18:45:55,960 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 18:45:55,983 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #7 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:45:56,026 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #7 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:46:06,474 | test | INFO | pool-9-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.00840014 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+------------+--------------+ 2021-05-17 18:46:06,499 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:46:16,572 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:46:16,598 | test | INFO | pool-9-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-17 18:46:16,890 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 18:47:06,042 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 18:47:06,289 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 830606256 | 117419148 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 271035856 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 1270736736 | 83289792 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 18:47:06,289 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 18:47:42,075 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:47:42,094 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:48:04,503 | test | INFO | pool-9-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........*.............**............................................................................*X | 25045.635 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:48:04,611 | test | INFO | pool-9-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **......................**.............................................................................*X | 99.916 | +-------+-----------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 18:48:05,812 | test | INFO | pool-9-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.....**.....................................................................*..........................*X | 126.020 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 18:48:06,094 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.9464166109 | 6437847040 | 3988594688 | 2763276288 / 7511588864 | 78742 / 228447 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8968300191 | 6437847040 | 4001796096 | 2772152320 / 7511588864 | 76804 / 219304 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9468332558 | 6437847040 | 3480530944 | 3444826112 / 9530753024 | 78436 / 226522 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 100 | 6437847040 | 4044464128 | 2756743168 / 7511588864 | 78942 / 217010 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 18:48:18,769 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:49:30,569 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 216282720 | 318377878 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 317256800 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 862736704 | 600720384 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 18:49:30,572 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #7 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:49:38,901 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 18:49:40,493 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 18:49:40,493 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before rebalance_in_out 2021-05-17 18:49:40,494 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 18:49:48,918 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 18:49:49,102 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-17 18:50:05,346 | test | INFO | pool-9-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 88.0240107004 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 92.8076859367 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 99.3791691434 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 99.4849642798 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:50:05,371 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:50:06,882 | test | INFO | pool-9-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........*........................................................................................X | 27112.363 | +-------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 18:50:07,150 | test | INFO | pool-9-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5211.290 | | 251.398 | ...X | 251.398 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:50:09,013 | test | INFO | pool-9-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 2551.119 | | 2413.133 | .............................................................................................X | 2413.133 | | 405.521 | ..............X | 405.521 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 18:50:11,148 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.18 2021-05-17 18:50:16,529 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 4.14 2021-05-17 18:50:21,667 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 7.94 2021-05-17 18:50:27,282 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.58 2021-05-17 18:50:32,384 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.19 2021-05-17 18:50:37,496 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 21.17 2021-05-17 18:50:42,690 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 25.49 2021-05-17 18:50:47,818 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-17 18:50:52,898 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:50:57,963 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:51:03,039 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 18:51:08,112 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-17 18:51:13,200 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 44.48 2021-05-17 18:51:18,480 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 49.97 2021-05-17 18:51:23,572 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 55.57 2021-05-17 18:51:28,655 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 60.83 2021-05-17 18:51:33,757 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:51:38,821 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:51:43,924 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 18:51:49,025 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 67.89 2021-05-17 18:51:54,144 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 71.38 2021-05-17 18:51:59,273 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 74.84 2021-05-17 18:52:04,433 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.99 2021-05-17 18:52:09,608 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.33 2021-05-17 18:52:14,684 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 84.66 2021-05-17 18:52:19,782 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 87.76 2021-05-17 18:52:24,862 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 90.91 2021-05-17 18:52:29,963 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 94.11 2021-05-17 18:52:35,036 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 96.67 2021-05-17 18:52:40,115 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:52:45,177 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 18:52:55,266 | test | INFO | pool-9-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:52:55,365 | test | INFO | pool-9-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 170.019000053 sec 2021-05-17 18:53:05,917 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 18:54:11,184 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 18:54:13,226 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 18:54:14,194 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 18:54:14,196 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 18:54:30,898 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 94% 2021-05-17 18:54:30,898 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 83% 2021-05-17 18:54:54,063 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 18:54:54,063 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 18:55:55,861 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1289999485 seconds to complete 2021-05-17 18:55:55,875 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 778.704s OK During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_7 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_8 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '1', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_8', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 8, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 18:56:05,374 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #8 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:56:06,059 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #8 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:56:06,533 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x289cfd type=client open_count=1 channel=[id: 0xfba3baca, 0.0.0.0/0.0.0.0:47482] timeout=300.0>) 2021-05-17 18:56:27,292 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x289cfe type=client open_count=1 channel=[id: 0xc5a585af, 0.0.0.0/0.0.0.0:52314] timeout=300.0>) 2021-05-17 18:56:29,311 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x289cff type=client open_count=1 channel=[id: 0x43238afb, 0.0.0.0/0.0.0.0:47703] timeout=300.0>) 2021-05-17 18:56:31,329 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x289d00 type=client open_count=1 channel=[id: 0x56748520, 0.0.0.0/0.0.0.0:47169] timeout=300.0>) 2021-05-17 18:56:55,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x289d01 type=client open_count=1 channel=[id: 0x47e0760f, 0.0.0.0/0.0.0.0:46632] timeout=300.0>) 2021-05-17 18:56:57,252 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x289d02 type=client open_count=1 channel=[id: 0x1350ab6d, 0.0.0.0/0.0.0.0:58566] timeout=300.0>) 2021-05-17 18:56:59,270 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x289d03 type=client open_count=1 channel=[id: 0x2eb5abb7, 0.0.0.0/0.0.0.0:52908] timeout=300.0>) 2021-05-17 18:57:23,346 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x289d04 type=client open_count=1 channel=[id: 0x462f539e, 0.0.0.0/0.0.0.0:54597] timeout=300.0>) 2021-05-17 18:57:25,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x289d05 type=client open_count=1 channel=[id: 0x47bd6d91, 0.0.0.0/0.0.0.0:42673] timeout=300.0>) 2021-05-17 18:57:27,388 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x289d06 type=client open_count=1 channel=[id: 0xcab65386, 0.0.0.0/0.0.0.0:46605] timeout=300.0>) 2021-05-17 18:57:51,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x289d07 type=client open_count=1 channel=[id: 0xce59d8d9, 0.0.0.0/0.0.0.0:39021] timeout=300.0>) 2021-05-17 18:57:53,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x289d08 type=client open_count=1 channel=[id: 0x277e30ab, 0.0.0.0/0.0.0.0:33055] timeout=300.0>) 2021-05-17 18:57:55,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x289d09 type=client open_count=1 channel=[id: 0x41dcc8c1, 0.0.0.0/0.0.0.0:53259] timeout=300.0>) 2021-05-17 18:58:19,953 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x289d0a type=client open_count=1 channel=[id: 0x502f597d, 0.0.0.0/0.0.0.0:57108] timeout=300.0>) 2021-05-17 18:58:21,970 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x289d0b type=client open_count=1 channel=[id: 0xf2c1af41, 0.0.0.0/0.0.0.0:51851] timeout=300.0>) 2021-05-17 18:58:23,989 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 18:58:41,176 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 18:58:55,391 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 18:58:55,417 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #8 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:58:55,460 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #8 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 18:59:05,586 | test | INFO | pool-10-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 14.7697538374 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 18:59:05,612 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 18:59:15,700 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 18:59:15,726 | test | INFO | pool-10-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1410000324 sec 2021-05-17 18:59:15,983 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 19:00:05,960 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 19:00:06,234 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 838860800 | 812183856 | 119761413 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 838860800 | 287921952 | 136 | | default | couchbase | 3 | none | 0 | 0 | 8388608000 | 1234957712 | 80817070 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 19:00:06,236 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 19:00:43,726 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:00:43,740 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:01:07,122 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.................................................................**..................*.................*X | 91.066 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:01:07,398 | test | INFO | pool-10-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *.................**...................................................................................**X | 72.750 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:01:09,269 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......**......................................................................*........................*X | 23965.855 | | 23920.421 | ..................................................................................................X | 23920.421 | +-----------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:01:09,576 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.9481683359 | 6437847040 | 3979743232 | 2753908736 / 7511588864 | 125653 / 366691 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.871006152 | 6437847040 | 4067758080 | 2716446720 / 7511588864 | 124186 / 243787 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9485733245 | 6437847040 | 3500912640 | 3435425792 / 9530753024 | 126087 / 273641 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9468332558 | 6437847040 | 4082151424 | 2730377216 / 7511588864 | 124359 / 265522 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 19:01:22,638 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:02:34,990 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 211119248 | 310520215 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 838860800 | 316486688 | 136 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 875220848 | 609183242 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 19:02:34,992 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #8 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:02:43,249 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-17 19:02:43,250 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-17 19:02:46,815 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:02:47,015 | test | INFO | pool-10-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 17.1869531159 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 17.8127395753 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 20.9890329678 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 12.7783527526 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:02:47,107 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:02:52,532 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:02:56,994 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:02:58,079 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:03:03,414 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.26 2021-05-17 19:03:09,592 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.44 2021-05-17 19:03:15,456 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 1.02 2021-05-17 19:03:15,803 | test | INFO | pool-10-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 2619.036 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:03:16,739 | test | INFO | pool-10-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 2152.209 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:03:16,963 | test | INFO | pool-10-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 13397.282 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:03:20,618 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 2.72 2021-05-17 19:03:25,755 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 4.66 2021-05-17 19:03:30,878 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 6.72 2021-05-17 19:03:36,068 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 8.97 2021-05-17 19:03:41,244 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 11.28 2021-05-17 19:03:46,351 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 14.31 2021-05-17 19:03:51,450 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 17.5 2021-05-17 19:03:56,569 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 20.94 2021-05-17 19:04:01,663 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 24.47 2021-05-17 19:04:06,802 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 27.98 2021-05-17 19:04:11,938 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 31.09 2021-05-17 19:04:17,020 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:04:22,085 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:04:27,335 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:04:32,632 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 36.12 2021-05-17 19:04:37,750 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 40.87 2021-05-17 19:04:42,862 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 45.72 2021-05-17 19:04:47,940 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 50.56 2021-05-17 19:04:53,062 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 55.48 2021-05-17 19:04:58,168 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 59.93 2021-05-17 19:05:03,269 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 63.94 2021-05-17 19:05:08,398 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:05:13,464 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:05:18,542 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:05:23,713 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 69.24 2021-05-17 19:05:28,816 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 72.1 2021-05-17 19:05:33,931 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 74.94 2021-05-17 19:05:39,013 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 77.93 2021-05-17 19:05:44,085 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 80.83 2021-05-17 19:05:49,177 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 83.83 2021-05-17 19:05:54,276 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 86.63 2021-05-17 19:05:59,385 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 89.53 2021-05-17 19:06:04,473 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 92.36 2021-05-17 19:06:09,648 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 95.24 2021-05-17 19:06:14,763 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 98.05 2021-05-17 19:06:19,841 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:06:24,903 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:06:31,210 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:06:41,477 | test | INFO | pool-10-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:06:41,644 | test | INFO | pool-10-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 234.628999949 sec 2021-05-17 19:06:50,016 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:07:39,989 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 19:07:39,990 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 19:07:56,142 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 75% 2021-05-17 19:07:56,144 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 85% 2021-05-17 19:08:15,604 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 19:08:15,605 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 19:09:11,253 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 19:09:11,269 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 795.321s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_8 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_9 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_9', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 9, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 19:09:20,630 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #9 test_data_load_collections_with_rebalance_in ========= 2021-05-17 19:09:21,329 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #9 test_data_load_collections_with_rebalance_in ========= 2021-05-17 19:09:21,759 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2e680e type=client open_count=1 channel=[id: 0x8480a2c5, 0.0.0.0/0.0.0.0:50049] timeout=300.0>) 2021-05-17 19:09:42,503 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2e680f type=client open_count=1 channel=[id: 0xe65885f0, 0.0.0.0/0.0.0.0:43012] timeout=300.0>) 2021-05-17 19:09:44,523 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2e6810 type=client open_count=1 channel=[id: 0xcb91f864, 0.0.0.0/0.0.0.0:34411] timeout=300.0>) 2021-05-17 19:09:46,540 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2e6811 type=client open_count=1 channel=[id: 0x75587475, 0.0.0.0/0.0.0.0:52414] timeout=300.0>) 2021-05-17 19:10:10,131 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2e6812 type=client open_count=1 channel=[id: 0x94e6f5cf, 0.0.0.0/0.0.0.0:55934] timeout=300.0>) 2021-05-17 19:10:12,154 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2e6813 type=client open_count=1 channel=[id: 0xd1bacf13, 0.0.0.0/0.0.0.0:53922] timeout=300.0>) 2021-05-17 19:10:14,173 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2e6814 type=client open_count=1 channel=[id: 0xa37db08b, 0.0.0.0/0.0.0.0:57991] timeout=300.0>) 2021-05-17 19:10:38,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2e6815 type=client open_count=1 channel=[id: 0x6d9d83ba, 0.0.0.0/0.0.0.0:51918] timeout=300.0>) 2021-05-17 19:10:40,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2e6816 type=client open_count=1 channel=[id: 0xdf8ff030, 0.0.0.0/0.0.0.0:57957] timeout=300.0>) 2021-05-17 19:10:42,612 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2e6817 type=client open_count=1 channel=[id: 0x0ac3cc23, 0.0.0.0/0.0.0.0:40807] timeout=300.0>) 2021-05-17 19:11:06,568 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2e6818 type=client open_count=1 channel=[id: 0x4baa5a96, 0.0.0.0/0.0.0.0:38942] timeout=300.0>) 2021-05-17 19:11:08,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2e6819 type=client open_count=1 channel=[id: 0xc28de5a9, 0.0.0.0/0.0.0.0:41676] timeout=300.0>) 2021-05-17 19:11:10,604 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2e681a type=client open_count=1 channel=[id: 0x33e9bd9c, 0.0.0.0/0.0.0.0:52244] timeout=300.0>) 2021-05-17 19:11:34,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2e681b type=client open_count=1 channel=[id: 0x81e02888, 0.0.0.0/0.0.0.0:52653] timeout=300.0>) 2021-05-17 19:11:36,681 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2e681c type=client open_count=1 channel=[id: 0xffcb8eac, 0.0.0.0/0.0.0.0:36489] timeout=300.0>) 2021-05-17 19:11:38,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 19:11:56,081 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 19:12:09,780 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 19:12:09,802 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #9 test_data_load_collections_with_rebalance_in ========= 2021-05-17 19:12:09,839 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #9 test_data_load_collections_with_rebalance_in ========= 2021-05-17 19:12:16,322 | test | INFO | pool-11-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 16.0688689623 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:12:16,348 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:12:26,438 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:12:26,463 | test | INFO | pool-11-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1400001049 sec 2021-05-17 19:12:26,710 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 19:13:06,247 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 19:13:06,473 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 622713464 | 75368204 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 629145600 | 205938192 | 102 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 919392760 | 44112721 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 19:13:06,473 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 19:13:32,536 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:13:32,546 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:14:30,312 | test | INFO | pool-11-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *......*...**X | 7.498 | | 6.000 | ......***...............................................................*..............................**X | 76.355 | | 64.631 | ...................................................................................***..........***X | 72.024 | | 67.002 | ......................................................................................X | 67.002 | +--------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:14:31,644 | test | INFO | pool-11-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......*............***..................................................................................**X | 9448.491 | | 9160.547 | ...............................................................................................***X | 9160.547 | | 8844.174 | ............................................................................................***.***X | 8929.074 | | 8650.186 | ..........................................................................................*X | 8650.186 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:14:31,869 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.................................................***..............*.......***...............**X | 69.039 | | 48.517 | ...........................................................****........................................*X | 80.603 | +--------+-----------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:14:32,154 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.9749991666 | 6437847040 | 3974586368 | 2757681152 / 7511588864 | 158521 / 313631 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.7133333333 | 6437847040 | 3995262976 | 2769838080 / 7511588864 | 157369 / 314666 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.6606202458 | 6437847040 | 3514347520 | 3425726464 / 9530753024 | 157637 / 208690 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 19:14:52,489 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:15:45,977 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 161064392 | 247099977 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 629145600 | 239207344 | 102 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 536493160 | 419530649 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 19:15:45,979 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #9 test_data_load_collections_with_rebalance_in ========= 2021-05-17 19:15:52,247 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in 2021-05-17 19:15:52,249 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-17 19:15:52,305 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:15:58,815 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:16:01,997 | test | INFO | pool-11-thread-30 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 17.4227525749 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 21.6946384227 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 19.3756458549 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:16:02,118 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:16:08,404 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:16:09,022 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1085 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:09,023 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:09,400 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-125 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:09,401 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-125 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:09,701 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1264 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1221 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:09,703 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1264 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1221 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,536 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1553 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,536 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1553 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,740 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,742 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-897 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,880 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:10,881 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-487 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,086 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1754 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,088 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1622 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1673 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,220 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1091 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1026 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1048 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1101 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,221 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1091 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1026 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1048 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1101 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,609 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1943 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,611 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1943 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,789 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:11,789 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,071 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-973 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,072 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-973 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,200 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2118 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2088 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,201 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2118 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2129 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2088 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,470 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2320 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,470 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1091 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1101 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,470 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2320 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,471 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1091 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1101 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,779 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,779 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1673 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,878 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,880 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2542 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,891 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:12,891 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1308 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1298 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:13,239 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1449 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:13,240 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1449 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:13,240 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:13,240 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2668 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:13,713 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.91 2021-05-17 19:16:14,134 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-79 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,138 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-79 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-162 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,405 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1859 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,407 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1859 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,575 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,576 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-162 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,653 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-213 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,654 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-213 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-220 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-383 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,910 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2149 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:14,911 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2149 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2129 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,065 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,065 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,065 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-552 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,066 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-220 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,255 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2149 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,256 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2149 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2129 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,667 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-766 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,667 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-766 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-610 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-698 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-780 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,710 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2842 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:15,710 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2842 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,059 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,061 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-552 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,157 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,158 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-922 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,351 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3085 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,351 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,493 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,496 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-698 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,617 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1152 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,618 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1152 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,841 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-922 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,842 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-922 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,938 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1368 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:16,940 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1368 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,099 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1152 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,101 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1152 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,266 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,267 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1551 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,381 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1368 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,381 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1368 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1223 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,608 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,609 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1620 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1613 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1783 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,690 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,691 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1551 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,907 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:17,907 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:18,881 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 1.84 2021-05-17 19:16:18,993 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:18,994 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2373 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,299 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2432 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,301 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2432 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,463 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,463 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2548 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,693 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,694 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1620 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1613 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1783 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,713 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,713 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2606 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2796 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,749 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,750 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2606 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2796 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,750 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3425 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3427 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3460 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:19,752 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3425 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3427 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3460 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,010 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1872 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,012 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1870 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,061 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2149 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2002 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,062 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2149 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2002 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,500 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2085 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,502 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2115 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2085 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2131 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,607 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,608 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2358 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,823 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,825 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2358 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2339 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2251 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,986 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:20,987 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,161 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,161 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,437 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,437 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2667 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,691 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,693 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2661 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,723 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2958 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,724 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2933 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2958 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,769 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:21,770 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,101 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3924 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3967 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,104 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3817 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3987 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3924 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3979 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3967 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,367 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,368 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,480 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,480 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,630 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,631 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,740 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,740 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3221 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,877 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,877 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3553 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,878 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-90 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-140 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:22,878 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3553 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,119 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4280 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4353 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4310 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,121 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4280 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4353 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4310 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,180 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,181 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,230 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,232 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,319 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,319 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,528 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,529 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-329 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-270 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,632 | test | INFO | pool-11-thread-17 | [table_view:display:72] Keys failed in bucket1:scope1:Cn_92iInR9lHM-26-786000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,632 | test | INFO | pool-11-thread-11 | [table_view:display:72] Keys failed in bucket1:scope1:Gg_BU51Nsiob5M8Y-26-787000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,986 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:23,987 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3534 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,035 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 3.08 2021-05-17 19:16:24,269 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,269 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,270 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,270 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,276 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,278 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,326 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,328 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,516 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,516 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,809 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,811 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,812 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,813 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,815 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,815 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4020 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,855 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,857 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-655 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:24,868 | test | INFO | pool-11-thread-29 | [table_view:display:72] Keys failed in bucket1:scope1:kGduR-26-787000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,016 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4991 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4872 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,017 | test | INFO | pool-11-thread-26 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:EhF-56-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,026 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 34 docs from 0 to 5000 2021-05-17 19:16:25,088 | test | INFO | pool-11-thread-29 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,088 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,089 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket1:scope1:collection1 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,099 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3837 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3872 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3961 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,101 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3957 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3837 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3872 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3961 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,279 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4203 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4355 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,280 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4203 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4355 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,280 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4203 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4355 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,282 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4203 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4355 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,282 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4203 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4355 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4246 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,282 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4393 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4203 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4355 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4246 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,285 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-958 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,286 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-957 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-958 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-912 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-920 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-882 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,437 | test | INFO | pool-11-thread-29 | [table_view:display:72] Keys failed in bucket1:scope2:um-26-791000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,437 | test | INFO | pool-11-thread-16 | [table_view:display:72] Keys failed in bucket1:_default:yYR6uuN33bdZL--26-781000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,437 | test | INFO | pool-11-thread-29 | [table_view:display:72] Keys failed in bucket1:scope2:um-26-791000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,437 | test | INFO | pool-11-thread-16 | [table_view:display:72] Keys failed in bucket1:_default:yYR6uuN33bdZL--26-781000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,512 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4065 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,513 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4065 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,684 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4427 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4571 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4454 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,686 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4427 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4571 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4454 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,687 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,687 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1174 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,710 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4427 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4571 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4454 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,711 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4427 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4571 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4454 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,720 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4427 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4571 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4454 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,720 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4427 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4571 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4454 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,831 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4326 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4363 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4355 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:25,832 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4326 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4363 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4355 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,443 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4991 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,443 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4991 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,444 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4991 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,444 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,444 | test | INFO | pool-11-thread-20 | [table_view:display:72] Keys failed in bucket1:B-57-148000:k-57-305000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,444 | test | INFO | pool-11-thread-23 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4801 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4991 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:26,446 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 34 docs from 0 to 5000 2021-05-17 19:16:27,651 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1237 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1272 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:27,651 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1237 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1272 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1244 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:27,769 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4427 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4417 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4454 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4411 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4587 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:27,770 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4581 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4427 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4417 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4454 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4411 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4587 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:28,339 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:28,341 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1457 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1458 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:28,855 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4765 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:28,855 | test | INFO | pool-11-thread-22 | [table_view:display:72] Keys failed in bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4765 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:29,200 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 4.7 2021-05-17 19:16:29,367 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1772 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:29,368 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1772 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:29,671 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 38 docs from 0 to 5000 2021-05-17 19:16:29,798 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:29,799 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1920 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:30,490 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2332 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:30,492 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2332 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:30,676 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 55 docs from 0 to 5000 2021-05-17 19:16:30,677 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 49 docs from 0 to 5000 2021-05-17 19:16:32,359 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:32,361 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3270 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3378 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:32,838 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:32,839 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3492 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3502 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:33,359 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3663 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:33,361 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3663 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:34,403 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 6.12 2021-05-17 19:16:34,470 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:34,471 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:34,934 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4334 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:34,934 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4358 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4334 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:35,940 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:35,940 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4746 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:36,380 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:36,381 | test | INFO | pool-11-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:uK6iONwZ-58-600000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4942 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:36,382 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 35 docs from 0 to 5000 2021-05-17 19:16:38,506 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:38,509 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-145 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:39,301 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:39,302 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-347 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:39,556 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 7.79 2021-05-17 19:16:39,743 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-499 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:39,744 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-435 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-499 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:40,596 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-765 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-776 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:40,598 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-765 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-776 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:41,036 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-844 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:41,036 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-844 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:41,575 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1006 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:41,576 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1006 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1196 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:42,036 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:42,038 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1260 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:42,566 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1715 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1685 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:42,569 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1731 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1715 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1685 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:44,756 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 9.52 2021-05-17 19:16:47,750 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4181 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4011 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:47,752 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4181 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4011 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:48,480 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:48,482 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4759 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4738 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4677 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:48,750 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4969 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:48,750 | test | INFO | pool-11-thread-3 | [table_view:display:72] Keys failed in bucket1:B-57-148000:HqK0-57-306000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4969 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:16:49,993 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 11.31 2021-05-17 19:16:54,786 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 19:16:54,786 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 19:16:54,789 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 19:16:54,789 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 19:16:54,789 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 15 to 17 2021-05-17 19:16:54,789 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 15 to 17 2021-05-17 19:16:54,790 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 15 to 17 2021-05-17 19:16:55,219 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 13.24 2021-05-17 19:17:00,410 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 14.25 2021-05-17 19:17:03,516 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 20 docs from 0 to 5000 2021-05-17 19:17:04,236 | test | INFO | pool-11-thread-25 | [table_view:display:72] Keys failed in bucket1:BEUnafNM65PD3Md-26-793000:cbmmRiq-0-26-796000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:04,280 | test | INFO | pool-11-thread-21 | [table_view:display:72] Keys failed in bucket1:BEUnafNM65PD3Md-26-793000:7wujf1Upu-KoNaodLdCB-26-796000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:04,302 | test | INFO | pool-11-thread-21 | [table_view:display:72] Keys failed in bucket1:_default:uvu4HDZJdTjWzy-26-784000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:04,357 | test | INFO | pool-11-thread-21 | [table_view:display:72] Keys failed in bucket1:scope2:tHM6JwYBp-26-790000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:04,358 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in bucket1:BEUnafNM65PD3Md-26-793000:lsrU7Z8-26-796000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:04,404 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in bucket1:BEUnafNM65PD3Md-26-793000:bMPcryJORtriplPJY-26-797000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:17:05,617 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 16.49 2021-05-17 19:17:08,016 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:08,016 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:08,017 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:08,019 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:08,019 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:08,019 | test | ERROR | pool-11-thread-24 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 19:17:10,839 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 17.41 2021-05-17 19:17:15,943 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 18.35 2021-05-17 19:17:21,066 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 20.01 2021-05-17 19:17:23,936 | test | INFO | pool-11-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+----------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***...**.........................................................***........................................*X | 1243.929 | | 1161.641 | ............................................................................................X | 1161.641 | | 973.470 | .............................................................................**X | 973.470 | | 559.210 | ...........................................*X | 559.210 | | 559.210 | ...........................................X | 559.210 | | 16.397 | ****.****X | 28.802 | +----------+----------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:17:25,582 | test | INFO | pool-11-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..............*...........................**..........................................................**.X | 4784.527 | | 4708.618 | .................................................................................................*X | 4708.618 | | 2700.636 | .......................................................**X | 2700.636 | | 2602.209 | .....................................................X | 2602.209 | | 2415.703 | .................................................***....................**...*..***................**X | 4395.442 | +----------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:17:26,022 | test | INFO | pool-11-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***.....**.............................................***.................................................*X | 1521.333 | | 698.570 | ............................................**X | 698.869 | | 297.853 | ..................X | 297.853 | | 292.063 | ..................X | 292.063 | | 292.063 | ..................X | 292.063 | | 290.773 | ..................**X | 295.747 | | 45.474 | .X | 45.474 | | 23.602 | **X | 23.602 | | 19.498 | **X | 19.498 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:17:26,178 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 22.21 2021-05-17 19:17:31,290 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 25.98 2021-05-17 19:17:36,403 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 29.29 2021-05-17 19:17:41,500 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 32.52 2021-05-17 19:17:46,654 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 34.08 2021-05-17 19:17:51,746 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 40.65 2021-05-17 19:17:56,848 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 47.22 2021-05-17 19:18:01,960 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 53.88 2021-05-17 19:18:07,082 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 59.71 2021-05-17 19:18:12,204 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 65.18 2021-05-17 19:18:17,359 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 67.23 2021-05-17 19:18:22,470 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 72.88 2021-05-17 19:18:27,572 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 77.76 2021-05-17 19:18:32,671 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 82.66 2021-05-17 19:18:37,828 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 87.14 2021-05-17 19:18:42,934 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 91.19 2021-05-17 19:18:48,043 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 94.46 2021-05-17 19:18:53,115 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 97.33 2021-05-17 19:18:58,796 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:19:08,905 | test | INFO | pool-11-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:19:08,944 | test | INFO | pool-11-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 186.946000099 sec 2021-05-17 19:19:09,035 | test | WARNING | Thread-25789 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:yYR6uuN33bdZL--26-781000 create failed keys from task: [u'test_collections-15'] 2021-05-17 19:19:09,036 | test | WARNING | Thread-25797 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:uvu4HDZJdTjWzy-26-784000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,045 | test | WARNING | Thread-25825 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ns2HzzcK-56-246000:Sh_uGirLP-56-437000 create failed keys from task: [u'test_collections-4020', u'test_collections-4581', u'test_collections-1026', u'test_collections-4571', u'test_collections-1101', u'test_collections-2796', u'test_collections-4454', u'test_collections-3246', u'test_collections-4499', u'test_collections-4411', u'test_collections-897', u'test_collections-2149', u'test_collections-4801', u'test_collections-4249', u'test_collections-1339', u'test_collections-2548', u'test_collections-3837', u'test_collections-4363', u'test_collections-4203', u'test_collections-4246', u'test_collections-1091', u'test_collections-907', u'test_collections-1048', u'test_collections-4991', u'test_collections-3221', u'test_collections-2373', u'test_collections-4355', u'test_collections-4393', u'test_collections-4427', u'test_collections-3534', u'test_collections-2129', u'test_collections-4509', u'test_collections-2606', u'test_collections-1673'] 2021-05-17 19:19:09,046 | test | WARNING | Thread-25830 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:BEUnafNM65PD3Md-26-793000:cbmmRiq-0-26-796000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,045 | test | WARNING | Thread-25826 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ns2HzzcK-56-246000:EhF-56-436000 create failed keys from task: [u'test_collections-2118', u'test_collections-1622', u'test_collections-1943', u'test_collections-3924', u'test_collections-3967', u'test_collections-1264', u'test_collections-1221', u'test_collections-3085', u'test_collections-4801', u'test_collections-2668', u'test_collections-3115', u'test_collections-2542', u'test_collections-4280', u'test_collections-3425', u'test_collections-3427', u'test_collections-3987', u'test_collections-4353', u'test_collections-4991', u'test_collections-1085', u'test_collections-3460', u'test_collections-4156', u'test_collections-4310', u'test_collections-4872', u'test_collections-4752', u'test_collections-3817', u'test_collections-2842', u'test_collections-1115', u'test_collections-2129', u'test_collections-1754', u'test_collections-3979', u'test_collections-2088', u'test_collections-2320', u'test_collections-1673', u'test_collections-1553'] 2021-05-17 19:19:09,092 | test | WARNING | Thread-25840 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:BEUnafNM65PD3Md-26-793000:bMPcryJORtriplPJY-26-797000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,095 | test | WARNING | Thread-25853 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:BEUnafNM65PD3Md-26-793000:7wujf1Upu-KoNaodLdCB-26-796000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,095 | test | WARNING | Thread-25849 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:BEUnafNM65PD3Md-26-793000:lsrU7Z8-26-796000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,108 | test | WARNING | Thread-25919 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:um-26-791000 create failed keys from task: [u'test_collections-15'] 2021-05-17 19:19:09,112 | test | WARNING | Thread-25863 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:B-57-148000:HqK0-57-306000 create failed keys from task: [u'test_collections-4181', u'test_collections-509', u'test_collections-844', u'test_collections-347', u'test_collections-765', u'test_collections-1006', u'test_collections-145', u'test_collections-4759', u'test_collections-4738', u'test_collections-1260', u'test_collections-4011', u'test_collections-4677', u'test_collections-1685', u'test_collections-435', u'test_collections-776', u'test_collections-499', u'test_collections-1731', u'test_collections-1715', u'test_collections-4969', u'test_collections-1196'] 2021-05-17 19:19:09,112 | test | WARNING | Thread-25864 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:Yjx8nvukKKhVUKp-56-672000:BEnMJnUP4oJO08Ni%JJr-56-925000 create failed keys from task: [u'test_collections-4581', u'test_collections-922', u'test_collections-1620', u'test_collections-4571', u'test_collections-4454', u'test_collections-1223', u'test_collections-3246', u'test_collections-1613', u'test_collections-4203', u'test_collections-2661', u'test_collections-869', u'test_collections-220', u'test_collections-1161', u'test_collections-2251', u'test_collections-3221', u'test_collections-2131', u'test_collections-4393', u'test_collections-1837', u'test_collections-4427', u'test_collections-2523', u'test_collections-2643', u'test_collections-2249', u'test_collections-2085', u'test_collections-1152', u'test_collections-1551', u'test_collections-4020', u'test_collections-2358', u'test_collections-2115', u'test_collections-162', u'test_collections-2752', u'test_collections-1783', u'test_collections-4499', u'test_collections-4411', u'test_collections-698', u'test_collections-4801', u'test_collections-4249', u'test_collections-3837', u'test_collections-4363', u'test_collections-4246', u'test_collections-708', u'test_collections-1368', u'test_collections-2339', u'test_collections-4991', u'test_collections-4355', u'test_collections-3534', u'test_collections-552', u'test_collections-4509', u'test_collections-1870', u'test_collections-1872'] 2021-05-17 19:19:09,112 | test | WARNING | Thread-25862 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:B-57-148000:k-57-305000 create failed keys from task: [u'test_collections-4581', u'test_collections-922', u'test_collections-766', u'test_collections-1620', u'test_collections-2958', u'test_collections-4571', u'test_collections-4454', u'test_collections-1223', u'test_collections-3246', u'test_collections-2149', u'test_collections-2667', u'test_collections-1613', u'test_collections-4203', u'test_collections-2661', u'test_collections-79', u'test_collections-220', u'test_collections-2933', u'test_collections-1161', u'test_collections-3221', u'test_collections-4393', u'test_collections-2523', u'test_collections-4427', u'test_collections-2643', u'test_collections-2249', u'test_collections-1152', u'test_collections-2002', u'test_collections-1551', u'test_collections-4020', u'test_collections-2358', u'test_collections-1941', u'test_collections-162', u'test_collections-2752', u'test_collections-1783', u'test_collections-4499', u'test_collections-4411', u'test_collections-2192', u'test_collections-610', u'test_collections-698', u'test_collections-4801', u'test_collections-213', u'test_collections-4249', u'test_collections-3837', u'test_collections-4363', u'test_collections-3553', u'test_collections-4246', u'test_collections-708', u'test_collections-1368', u'test_collections-780', u'test_collections-2339', u'test_collections-383', u'test_collections-4991', u'test_collections-4355', u'test_collections-3534', u'test_collections-552', u'test_collections-4509'] 2021-05-17 19:19:09,118 | test | WARNING | Thread-25920 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:uK6iONwZ-58-600000 create failed keys from task: [u'test_collections-3492', u'test_collections-329', u'test_collections-920', u'test_collections-882', u'test_collections-4133', u'test_collections-1582', u'test_collections-4334', u'test_collections-4052', u'test_collections-655', u'test_collections-90', u'test_collections-1412', u'test_collections-1457', u'test_collections-1458', u'test_collections-1174', u'test_collections-1772', u'test_collections-3270', u'test_collections-428', u'test_collections-3502', u'test_collections-1920', u'test_collections-140', u'test_collections-2332', u'test_collections-3663', u'test_collections-4358', u'test_collections-1244', u'test_collections-838', u'test_collections-957', u'test_collections-958', u'test_collections-912', u'test_collections-4746', u'test_collections-1237', u'test_collections-4942', u'test_collections-270', u'test_collections-1272', u'test_collections-1274', u'test_collections-3378'] 2021-05-17 19:19:09,121 | test | WARNING | Thread-25865 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:Yjx8nvukKKhVUKp-56-672000:D-56-926000 create failed keys from task: [u'test_collections-4581', u'test_collections-487', u'test_collections-125', u'test_collections-4417', u'test_collections-1308', u'test_collections-1101', u'test_collections-2796', u'test_collections-4454', u'test_collections-2432', u'test_collections-3246', u'test_collections-3961', u'test_collections-4499', u'test_collections-4411', u'test_collections-1859', u'test_collections-973', u'test_collections-2149', u'test_collections-2822', u'test_collections-4326', u'test_collections-3957', u'test_collections-1339', u'test_collections-2548', u'test_collections-3838', u'test_collections-3837', u'test_collections-4363', u'test_collections-3872', u'test_collections-1298', u'test_collections-4765', u'test_collections-1091', u'test_collections-1449', u'test_collections-4355', u'test_collections-517', u'test_collections-4427', u'test_collections-3534', u'test_collections-2129', u'test_collections-4509', u'test_collections-2606', u'test_collections-4065', u'test_collections-4587'] 2021-05-17 19:19:09,125 | test | WARNING | Thread-25881 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:kGduR-26-787000 update failed keys from task: [u'test_collections-3'] 2021-05-17 19:19:09,135 | test | WARNING | Thread-25875 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:collection1 update failed keys from task: [u'test_collections-3'] 2021-05-17 19:19:09,145 | test | WARNING | Thread-25875 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:collection1 create failed keys from task: [u'test_collections-15'] 2021-05-17 19:19:09,160 | test | WARNING | Thread-25925 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:tHM6JwYBp-26-790000 delete failed keys from task: [u'test_collections-2'] 2021-05-17 19:19:09,171 | test | WARNING | Thread-25895 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:Gg_BU51Nsiob5M8Y-26-787000 update failed keys from task: [u'test_collections-3'] 2021-05-17 19:19:09,184 | test | WARNING | Thread-25896 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:Cn_92iInR9lHM-26-786000 update failed keys from task: [u'test_collections-3'] 2021-05-17 19:19:21,482 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:20:41,131 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 19:20:41,131 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 19:20:57,025 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 19:20:57,026 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 19:21:17,530 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 19:21:17,532 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 19:22:07,986 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1300001144 seconds to complete 2021-05-17 19:22:08,002 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 776.791s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 9 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_9 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_10 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_10', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 10, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 19:22:17,494 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #10 test_data_load_collections_with_rebalance_out ========= 2021-05-17 19:22:18,207 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #10 test_data_load_collections_with_rebalance_out ========= 2021-05-17 19:22:18,654 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x34386c type=client open_count=1 channel=[id: 0x7a5e7815, 0.0.0.0/0.0.0.0:33821] timeout=300.0>) 2021-05-17 19:22:39,405 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x34386d type=client open_count=1 channel=[id: 0x62c51784, 0.0.0.0/0.0.0.0:56137] timeout=300.0>) 2021-05-17 19:22:41,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x34386e type=client open_count=1 channel=[id: 0x24798e0a, 0.0.0.0/0.0.0.0:50750] timeout=300.0>) 2021-05-17 19:22:43,443 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x34386f type=client open_count=1 channel=[id: 0x3fb6425d, 0.0.0.0/0.0.0.0:46523] timeout=300.0>) 2021-05-17 19:23:07,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x343870 type=client open_count=1 channel=[id: 0xa8311486, 0.0.0.0/0.0.0.0:34321] timeout=300.0>) 2021-05-17 19:23:09,785 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x343871 type=client open_count=1 channel=[id: 0xc14040d4, 0.0.0.0/0.0.0.0:51325] timeout=300.0>) 2021-05-17 19:23:11,803 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x343872 type=client open_count=1 channel=[id: 0xd4167a84, 0.0.0.0/0.0.0.0:47320] timeout=300.0>) 2021-05-17 19:23:36,102 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x343873 type=client open_count=1 channel=[id: 0xab844931, 0.0.0.0/0.0.0.0:55817] timeout=300.0>) 2021-05-17 19:23:38,119 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x343874 type=client open_count=1 channel=[id: 0xfc3ce941, 0.0.0.0/0.0.0.0:60737] timeout=300.0>) 2021-05-17 19:23:40,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x343875 type=client open_count=1 channel=[id: 0x125d6314, 0.0.0.0/0.0.0.0:44475] timeout=300.0>) 2021-05-17 19:24:04,437 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x343876 type=client open_count=1 channel=[id: 0xf961835b, 0.0.0.0/0.0.0.0:50997] timeout=300.0>) 2021-05-17 19:24:06,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x343877 type=client open_count=1 channel=[id: 0xd42045ae, 0.0.0.0/0.0.0.0:42080] timeout=300.0>) 2021-05-17 19:24:08,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x343878 type=client open_count=1 channel=[id: 0x5162fbdd, 0.0.0.0/0.0.0.0:33653] timeout=300.0>) 2021-05-17 19:24:32,532 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x343879 type=client open_count=1 channel=[id: 0xb13eb019, 0.0.0.0/0.0.0.0:57768] timeout=300.0>) 2021-05-17 19:24:34,549 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x34387a type=client open_count=1 channel=[id: 0x776b65b8, 0.0.0.0/0.0.0.0:35786] timeout=300.0>) 2021-05-17 19:24:36,568 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 19:24:54,003 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 19:25:08,372 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 19:25:08,397 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #10 test_data_load_collections_with_rebalance_out ========= 2021-05-17 19:25:08,434 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #10 test_data_load_collections_with_rebalance_out ========= 2021-05-17 19:25:22,951 | test | INFO | pool-12-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.41643890407 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:25:22,976 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:25:33,052 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:25:33,078 | test | INFO | pool-12-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1260001659 sec 2021-05-17 19:25:33,342 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 19:26:18,658 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 19:26:18,941 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 509450776 | 125789025 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 230376432 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 497923224 | 92334314 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 19:26:18,943 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 19:27:07,823 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:27:07,834 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:27:51,730 | test | INFO | pool-12-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 152.445 | | 78.104 | ..................................................*X | 78.104 | | 36.375 | ......................**X | 36.375 | | 22.506 | .............X | 22.506 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 19:27:53,546 | test | INFO | pool-12-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...............*.........................**........................................................***....*X | 12820.444 | | 12791.106 | ..................................................................................................*X | 12791.106 | +-----------+----------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:27:53,690 | test | INFO | pool-12-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **..................................................**...........................*..**.....................**X | 95.993 | | 63.914 | .................................................................*X | 63.914 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:27:54,065 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 97.7780750004 | 6437847040 | 4209164288 | 2543931392 / 7511588864 | 88728 / 178986 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.841426163 | 6437847040 | 4186832896 | 2552303616 / 7511588864 | 85846 / 151514 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.0084465663 | 6437847040 | 4235366400 | 2533969920 / 7511588864 | 87849 / 174605 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.0852640716 | 6437847040 | 3725029376 | 3219877888 / 9530753024 | 88224 / 173088 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9497613665 | 6437847040 | 4258799616 | 2536894464 / 7511588864 | 87739 / 174349 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 19:28:08,948 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:29:37,661 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 163289992 | 253450717 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 245651472 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 565295432 | 415906168 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 19:29:37,663 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #10 test_data_load_collections_with_rebalance_out ========= 2021-05-17 19:29:47,799 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_out 2021-05-17 19:29:47,799 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-17 19:29:47,865 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:29:48,020 | test | INFO | pool-12-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 9.97333333333 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 9.58333333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.74016233604 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.5516666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.53015883598 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:29:48,053 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:29:54,782 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:29:56,266 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:29:59,232 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,233 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-31 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,454 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,456 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-31 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,490 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,490 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-31 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,717 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-31 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,719 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-109 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-31 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,921 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:29:59,921 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,128 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.75 2021-05-17 19:30:00,173 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,174 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,180 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,181 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,191 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-813 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-983 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,193 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-983 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,441 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,441 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,513 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-813 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-983 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,513 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-983 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,517 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-813 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-983 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,519 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-983 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,823 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-813 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-983 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,825 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-813 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-983 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,829 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1259 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:00,831 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1259 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,053 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1259 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,055 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1259 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1248 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,055 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1259 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,056 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1259 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1248 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,170 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,171 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,308 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,309 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,328 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,328 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,400 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,401 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,494 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,496 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1248 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1204 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1394 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,543 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,543 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,572 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,573 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,796 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,796 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1576 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,980 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2069 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:01,982 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2069 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2025 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,161 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2069 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,161 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2069 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2025 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,170 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,171 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1676 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,180 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2069 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,181 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2069 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,611 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1842 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,611 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1842 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,713 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,713 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,917 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,917 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2025 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,933 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:02,934 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,062 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,063 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2757 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,290 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2332 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,292 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2332 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,430 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3367 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,430 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3367 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,683 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-208 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-398 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,684 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-208 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-398 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,780 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,782 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3415 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3585 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,785 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2440 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:03,786 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2440 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,020 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-464 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,022 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-464 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,101 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3614 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3784 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,104 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3614 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3784 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,358 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3916 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3886 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,359 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3916 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3886 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,901 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4322 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,901 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4322 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,914 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:04,915 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-879 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,151 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4450 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,153 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4450 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,463 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,463 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1233 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,477 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 1.26 2021-05-17 19:30:05,701 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,703 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1541 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,713 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4909 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4899 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4877 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,714 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4909 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4877 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4899 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,971 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1611 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1781 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:05,973 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1611 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1781 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,273 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1988 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,275 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1818 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1966 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1988 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,329 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,331 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,427 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3166 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,428 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3166 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,519 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2182 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2012 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,519 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2182 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2012 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,645 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,647 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-252 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-360 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,766 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,767 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,920 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,921 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-412 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-520 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-582 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,947 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:06,947 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,059 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3531 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,059 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3531 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,249 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3531 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,250 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3531 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,408 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,410 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-930 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,535 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,536 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,693 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,694 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,710 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1057 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,710 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1164 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1057 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,789 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,790 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,842 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 27 docs from 0 to 5000 2021-05-17 19:30:07,927 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,927 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,993 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:07,993 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1276 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,137 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,138 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,204 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,206 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,221 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,223 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1408 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1598 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1504 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1494 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,361 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,362 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3243 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,398 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,398 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4585 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,413 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,414 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4519 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4585 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4489 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,492 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1725 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,493 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1786 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1725 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1616 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,621 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4707 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4697 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,622 | test | INFO | pool-12-thread-25 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4707 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4697 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,638 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4707 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4697 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,641 | test | INFO | pool-12-thread-3 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:CRTTAT9O1-53-790000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4634 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4707 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4697 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,736 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3758 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,737 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3758 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,799 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 28 docs from 0 to 5000 2021-05-17 19:30:08,802 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 29 docs from 0 to 5000 2021-05-17 19:30:08,944 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2057 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,944 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2057 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,964 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:08,966 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3953 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,144 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,145 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-145 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,171 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,173 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2276 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2345 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,184 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,184 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4075 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4146 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,384 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2468 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,384 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2468 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2437 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2504 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2494 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,407 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,407 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4367 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,632 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,634 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4415 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4585 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,637 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2725 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,638 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2725 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,661 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,661 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-637 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,884 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,885 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2914 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2884 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,931 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,933 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,976 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4758 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:09,977 | test | INFO | pool-12-thread-7 | [table_view:display:72] Keys failed in bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4758 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,178 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,180 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2345 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,186 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 23 docs from 0 to 5000 2021-05-17 19:30:10,203 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,203 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,477 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2468 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,479 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2468 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,737 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 1.86 2021-05-17 19:30:10,809 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,812 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1649 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,865 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3762 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:10,865 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3762 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,289 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4139 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,289 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4139 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,290 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2209 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2399 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,292 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2209 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2399 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,661 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2659 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,663 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2659 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,704 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,704 | test | INFO | pool-12-thread-18 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4569 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,918 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2846 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2878 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:11,920 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2846 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2878 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:12,144 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3142 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:12,144 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3142 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3124 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,098 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2878 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,098 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2878 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,351 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,351 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3124 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,756 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3538 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:13,757 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3538 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,177 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3964 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,177 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3964 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,362 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4014 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,364 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4184 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4014 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,505 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3964 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,506 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3964 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,615 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4350 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,617 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4350 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,835 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4422 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,836 | test | INFO | pool-12-thread-2 | [table_view:display:72] Keys failed in bucket1:sH-53-552000:gDD-53-789000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4422 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,953 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4350 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:14,957 | test | INFO | pool-12-thread-30 | [table_view:display:72] Keys failed in bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4350 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:30:15,155 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 25 docs from 0 to 5000 2021-05-17 19:30:15,168 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 32 docs from 0 to 5000 2021-05-17 19:30:15,558 | test | ERROR | pool-12-thread-15 | [task:call:3809] Failed to load 14 docs from 0 to 5000 2021-05-17 19:30:16,058 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-05-17 19:30:21,230 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 2.84 2021-05-17 19:30:23,227 | test | INFO | pool-12-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 2281.814 | | 1152.208 | .................................................X | 1152.208 | | 810.285 | ..................................X | 810.285 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:30:23,539 | test | INFO | pool-12-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..........................................................................*.....................*.....X | 10128.936 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:30:23,681 | test | INFO | pool-12-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...........................................................................*...................*......X | 2040.747 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:30:26,434 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 5.33 2021-05-17 19:30:31,551 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.46 2021-05-17 19:30:36,665 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 10.84 2021-05-17 19:30:41,773 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 13.92 2021-05-17 19:30:46,894 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 17.17 2021-05-17 19:30:52,017 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 20.41 2021-05-17 19:30:57,105 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 23.66 2021-05-17 19:31:02,187 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 27.2 2021-05-17 19:31:07,292 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 30.8 2021-05-17 19:31:12,404 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:31:17,470 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:31:23,753 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.96 2021-05-17 19:31:28,855 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.9 2021-05-17 19:31:33,953 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 49.77 2021-05-17 19:31:39,055 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 55.24 2021-05-17 19:31:44,176 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 60.45 2021-05-17 19:31:49,266 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 64.88 2021-05-17 19:31:54,342 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:31:59,408 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:32:04,496 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 68.23 2021-05-17 19:32:09,614 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 73.8 2021-05-17 19:32:14,743 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 78.51 2021-05-17 19:32:19,844 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 82.37 2021-05-17 19:32:24,944 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 85.91 2021-05-17 19:32:30,049 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 89.48 2021-05-17 19:32:35,142 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 93.22 2021-05-17 19:32:40,269 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 96.98 2021-05-17 19:32:45,357 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:32:50,447 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:32:56,358 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:33:06,516 | test | INFO | pool-12-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:33:06,671 | test | INFO | pool-12-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 198.650000095 sec 2021-05-17 19:33:06,767 | test | WARNING | Thread-28675 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8tRkQy5yTtno1lNT4niX-52-927000:7TNN1ia9NZPKZ7t2_-53-143000 create failed keys from task: [u'test_collections-847', u'test_collections-3964', u'test_collections-145', u'test_collections-2659', u'test_collections-2878', u'test_collections-1649', u'test_collections-3142', u'test_collections-3124', u'test_collections-2399', u'test_collections-4350', u'test_collections-637', u'test_collections-2820', u'test_collections-2846', u'test_collections-2209'] 2021-05-17 19:33:06,767 | test | WARNING | Thread-28673 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:3i%Dg-52-631000 create failed keys from task: [u'test_collections-3886', u'test_collections-4899', u'test_collections-2757', u'test_collections-4450', u'test_collections-3367', u'test_collections-939', u'test_collections-3916', u'test_collections-813', u'test_collections-1576', u'test_collections-1259', u'test_collections-4322', u'test_collections-2025', u'test_collections-2069', u'test_collections-3631', u'test_collections-749', u'test_collections-109', u'test_collections-31', u'test_collections-983', u'test_collections-1204', u'test_collections-4877', u'test_collections-3784', u'test_collections-3585', u'test_collections-4909', u'test_collections-1676', u'test_collections-3614', u'test_collections-3415', u'test_collections-1394'] 2021-05-17 19:33:06,769 | test | WARNING | Thread-28676 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8tRkQy5yTtno1lNT4niX-52-927000:odq-53-142000 create failed keys from task: [u'test_collections-520', u'test_collections-1786', u'test_collections-2437', u'test_collections-4139', u'test_collections-1504', u'test_collections-360', u'test_collections-2914', u'test_collections-3762', u'test_collections-2276', u'test_collections-412', u'test_collections-930', u'test_collections-2468', u'test_collections-4569', u'test_collections-252', u'test_collections-2504', u'test_collections-1616', u'test_collections-1494', u'test_collections-1057', u'test_collections-2345', u'test_collections-38', u'test_collections-1408', u'test_collections-582', u'test_collections-1725', u'test_collections-1164', u'test_collections-2494', u'test_collections-2057', u'test_collections-3180', u'test_collections-1598', u'test_collections-2725', u'test_collections-3010', u'test_collections-1276', u'test_collections-2884'] 2021-05-17 19:33:06,779 | test | WARNING | Thread-28713 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:sH-53-552000:Hg1XEzvPKzYXiiOmOJZUb-56-29000 create failed keys from task: [u'test_collections-4415', u'test_collections-2757', u'test_collections-3166', u'test_collections-3243', u'test_collections-4697', u'test_collections-939', u'test_collections-813', u'test_collections-1576', u'test_collections-3953', u'test_collections-1259', u'test_collections-4489', u'test_collections-2025', u'test_collections-2069', u'test_collections-4367', u'test_collections-749', u'test_collections-109', u'test_collections-31', u'test_collections-983', u'test_collections-1248', u'test_collections-1204', u'test_collections-4634', u'test_collections-4519', u'test_collections-4075', u'test_collections-1676', u'test_collections-4707', u'test_collections-4585', u'test_collections-1394', u'test_collections-3531', u'test_collections-4146'] 2021-05-17 19:33:06,780 | test | WARNING | Thread-28714 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:sH-53-552000:CRTTAT9O1-53-790000 create failed keys from task: [u'test_collections-4415', u'test_collections-2757', u'test_collections-3243', u'test_collections-4697', u'test_collections-939', u'test_collections-813', u'test_collections-1576', u'test_collections-3953', u'test_collections-1259', u'test_collections-4489', u'test_collections-2069', u'test_collections-2025', u'test_collections-4367', u'test_collections-749', u'test_collections-109', u'test_collections-31', u'test_collections-983', u'test_collections-1204', u'test_collections-1248', u'test_collections-4634', u'test_collections-4519', u'test_collections-4075', u'test_collections-1676', u'test_collections-4707', u'test_collections-4585', u'test_collections-1394', u'test_collections-3531', u'test_collections-4146'] 2021-05-17 19:33:06,767 | test | WARNING | Thread-28674 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:xo7uDOQ-m1W0i05PYfO7-52-520000:Jzo-52-632000 create failed keys from task: [u'test_collections-4415', u'test_collections-3243', u'test_collections-813', u'test_collections-1576', u'test_collections-3953', u'test_collections-3758', u'test_collections-2025', u'test_collections-4367', u'test_collections-749', u'test_collections-109', u'test_collections-31', u'test_collections-983', u'test_collections-4758', u'test_collections-1248', u'test_collections-1204', u'test_collections-1842', u'test_collections-2332', u'test_collections-4075', u'test_collections-1676', u'test_collections-4585', u'test_collections-1394', u'test_collections-2440', u'test_collections-4146'] 2021-05-17 19:33:06,780 | test | WARNING | Thread-28715 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:sH-53-552000:gDD-53-789000 create failed keys from task: [u'test_collections-4184', u'test_collections-208', u'test_collections-3964', u'test_collections-2878', u'test_collections-1988', u'test_collections-1781', u'test_collections-3124', u'test_collections-1541', u'test_collections-4014', u'test_collections-1818', u'test_collections-2820', u'test_collections-2468', u'test_collections-1611', u'test_collections-371', u'test_collections-2345', u'test_collections-2182', u'test_collections-464', u'test_collections-1966', u'test_collections-2012', u'test_collections-4350', u'test_collections-879', u'test_collections-398', u'test_collections-3538', u'test_collections-1233', u'test_collections-4422'] 2021-05-17 19:33:15,099 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:34:06,210 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 19:34:06,211 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 19:34:21,930 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 19:34:21,931 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 19:34:46,655 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 19:34:46,655 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 19:35:40,970 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1380000114 seconds to complete 2021-05-17 19:35:40,986 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete During the test, ok Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 ---------------------------------------------------------------------- Ran 1 test in 812.880s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 10 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_10 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_11', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 11, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 19:35:50,372 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #11 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 19:35:51,095 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #11 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 19:35:51,528 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3a0bad type=client open_count=1 channel=[id: 0xa3564346, 0.0.0.0/0.0.0.0:55884] timeout=300.0>) 2021-05-17 19:36:12,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3a0bae type=client open_count=1 channel=[id: 0x482f4f79, 0.0.0.0/0.0.0.0:48295] timeout=300.0>) 2021-05-17 19:36:14,316 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3a0baf type=client open_count=1 channel=[id: 0x7c16abed, 0.0.0.0/0.0.0.0:57116] timeout=300.0>) 2021-05-17 19:36:16,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3a0bb0 type=client open_count=1 channel=[id: 0xfa1beff9, 0.0.0.0/0.0.0.0:51971] timeout=300.0>) 2021-05-17 19:36:40,115 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3a0bb1 type=client open_count=1 channel=[id: 0x7c5ccec4, 0.0.0.0/0.0.0.0:48837] timeout=300.0>) 2021-05-17 19:36:42,134 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3a0bb2 type=client open_count=1 channel=[id: 0x1db6352f, 0.0.0.0/0.0.0.0:39494] timeout=300.0>) 2021-05-17 19:36:44,153 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3a0bb3 type=client open_count=1 channel=[id: 0xdeb88937, 0.0.0.0/0.0.0.0:44643] timeout=300.0>) 2021-05-17 19:37:08,108 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3a0bb4 type=client open_count=1 channel=[id: 0xe6f63caf, 0.0.0.0/0.0.0.0:56572] timeout=300.0>) 2021-05-17 19:37:10,127 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3a0bb5 type=client open_count=1 channel=[id: 0xdb005091, 0.0.0.0/0.0.0.0:58813] timeout=300.0>) 2021-05-17 19:37:12,144 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3a0bb6 type=client open_count=1 channel=[id: 0xc30b56cc, 0.0.0.0/0.0.0.0:53034] timeout=300.0>) 2021-05-17 19:37:36,095 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3a0bb7 type=client open_count=1 channel=[id: 0x063e0e4d, 0.0.0.0/0.0.0.0:57302] timeout=300.0>) 2021-05-17 19:37:38,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3a0bb8 type=client open_count=1 channel=[id: 0xc1a9d5af, 0.0.0.0/0.0.0.0:50696] timeout=300.0>) 2021-05-17 19:37:40,140 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3a0bb9 type=client open_count=1 channel=[id: 0xb8392dc5, 0.0.0.0/0.0.0.0:39387] timeout=300.0>) 2021-05-17 19:38:04,203 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3a0bba type=client open_count=1 channel=[id: 0x5b8227f4, 0.0.0.0/0.0.0.0:38238] timeout=300.0>) 2021-05-17 19:38:06,220 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3a0bbb type=client open_count=1 channel=[id: 0x4970bd67, 0.0.0.0/0.0.0.0:43361] timeout=300.0>) 2021-05-17 19:38:08,240 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 19:38:25,234 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 19:38:39,016 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 19:38:39,038 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #11 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 19:38:39,085 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #11 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 19:38:49,828 | test | INFO | pool-13-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.36184539485 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:38:49,851 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:38:59,934 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:38:59,971 | test | INFO | pool-13-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1449999809 sec 2021-05-17 19:39:00,306 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 19:39:45,792 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 19:39:46,105 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 601566784 | 119864073 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 838860800 | 209890112 | 136 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 781314800 | 89619406 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 19:39:46,107 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 19:40:29,078 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:40:29,086 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:41:16,459 | test | INFO | pool-13-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **....................................................................................................**X | 25.494 | | 0.000 | *******X | 0.000 | +-------+-----------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 19:41:17,263 | test | INFO | pool-13-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 166.459 | | 38.995 | ......................***..................**X | 69.016 | | 0.000 | **X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 19:41:18,167 | test | INFO | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **...............................................***...............................................***.....***.*X | 11416.698 | | 11261.443 | .................................................................................................X | 11261.443 | +-----------+-------------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:41:18,528 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.844759377 | 6437847040 | 4121911296 | 2609201152 / 7511588864 | 98186 / 196026 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8421612282 | 6437847040 | 4173213696 | 2565066752 / 7511588864 | 97511 / 196473 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.5041349818 | 6437847040 | 3610701824 | 3329609728 / 9530753024 | 98234 / 182477 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9733333333 | 6437847040 | 4197814272 | 2579668992 / 7511588864 | 125349 / 251811 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 19:41:30,470 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:42:45,515 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 159176768 | 233631525 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 838860800 | 245375840 | 136 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 522655616 | 403304698 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 19:42:45,516 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #11 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 19:42:53,871 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during swap_rebalance 2021-05-17 19:42:53,871 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-17 19:42:57,427 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:42:57,619 | test | INFO | pool-13-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 17.0302838381 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 14.5316666667 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.9686328105 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.94983416943 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:42:57,680 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:43:02,838 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:43:07,974 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:43:08,121 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.19 2021-05-17 19:43:11,625 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-648 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:11,627 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-648 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-745 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:12,069 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:12,071 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-839 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:12,821 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:12,822 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1273 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,003 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,003 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,005 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-745 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,005 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-745 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,240 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1562 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,242 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1501 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1491 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1562 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:13,338 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.79 2021-05-17 19:43:14,040 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1972 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:14,042 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1972 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:14,744 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:14,746 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1605 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1795 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:14,749 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:14,750 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1605 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1795 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:15,345 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1108 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1098 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:15,345 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1108 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1098 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1177 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,436 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2284 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,437 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2284 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2378 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,440 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2284 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,440 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2284 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2378 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,457 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,459 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,671 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,673 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,897 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,898 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,901 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:17,901 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,371 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,372 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,404 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2128 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,405 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2128 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,480 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,482 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,483 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,483 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,655 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,657 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,704 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2284 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,706 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2314 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2284 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2378 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,779 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 1.59 2021-05-17 19:43:18,802 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,802 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,809 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,811 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3137 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,983 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:18,986 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2466 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,344 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2647 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,345 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2647 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,470 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,471 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,519 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,519 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,546 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,548 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3652 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,549 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1135 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,549 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1159 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1135 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,654 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2824 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,655 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2824 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,749 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,750 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,778 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,779 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,869 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,871 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,956 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3120 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:19,957 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3120 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,038 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,039 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,072 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,072 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,073 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,075 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4024 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,311 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,312 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,332 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,332 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,342 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,342 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,579 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,582 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,582 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,582 | test | INFO | pool-13-thread-5 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,588 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,588 | test | INFO | pool-13-thread-4 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,658 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:20,660 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:21,969 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 19 docs from 0 to 5000 2021-05-17 19:43:22,832 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:22,832 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,101 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-516 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,101 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-516 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-486 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,520 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-916 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-886 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,522 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-916 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-886 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,773 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3326 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:23,775 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3326 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:24,323 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:24,325 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:6h88kg-4-632000:-%TfYp-4-899000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3879 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:24,391 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 15 docs from 0 to 5000 2021-05-17 19:43:24,394 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 15 docs from 0 to 5000 2021-05-17 19:43:24,398 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 13 docs from 0 to 5000 2021-05-17 19:43:24,901 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 1.9 2021-05-17 19:43:25,428 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 19:43:26,088 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:26,088 | test | INFO | pool-13-thread-21 | [table_view:display:72] Keys failed in bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3469 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:28,510 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-896 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:28,513 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-896 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:28,992 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:28,992 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1330 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:29,266 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1553 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:29,267 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1442 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1553 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:30,292 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 2.21 2021-05-17 19:43:30,536 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3068 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:30,536 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3068 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:30,832 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3377 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:30,832 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3377 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:31,052 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3405 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3595 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:31,052 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3405 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3595 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:31,598 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 6 docs from 0 to 5000 2021-05-17 19:43:32,532 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4425 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:32,532 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4425 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:32,740 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4642 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:32,742 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4642 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:33,012 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4835 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:33,013 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:43:33,013 | test | ERROR | pool-13-thread-18 | [task:call:3809] Failed to load 12 docs from 0 to 5000 2021-05-17 19:43:35,640 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 2.68 2021-05-17 19:43:39,635 | test | INFO | pool-13-thread-9 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................*X | 2107.786 | | 1292.547 | ............................................................X | 1292.547 | | 245.958 | ..........X | 245.958 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:43:40,926 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 3.96 2021-05-17 19:43:41,418 | test | INFO | pool-13-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.....................................**..............................................................*.X | 2015.252 | | 897.317 | ...........................................X | 897.317 | +---------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:43:42,144 | test | INFO | pool-13-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.......................................................**....................................*......*...X | 8222.363 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:43:46,032 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 6.05 2021-05-17 19:43:51,140 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 8.15 2021-05-17 19:43:56,256 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 10.57 2021-05-17 19:44:01,349 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 13.68 2021-05-17 19:44:06,479 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 17.19 2021-05-17 19:44:11,621 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 20.51 2021-05-17 19:44:16,736 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 23.6 2021-05-17 19:44:21,825 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 26.79 2021-05-17 19:44:26,911 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 29.99 2021-05-17 19:44:32,022 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 33.02 2021-05-17 19:44:37,092 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:44:42,157 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:44:47,236 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 35.48 2021-05-17 19:44:52,341 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 40.54 2021-05-17 19:44:57,436 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 45.53 2021-05-17 19:45:02,536 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 50.71 2021-05-17 19:45:07,674 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 55.78 2021-05-17 19:45:12,762 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 59.82 2021-05-17 19:45:17,890 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 63.85 2021-05-17 19:45:22,986 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:45:28,052 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:45:35,351 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:45:40,454 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 71.09 2021-05-17 19:45:45,559 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 74.3 2021-05-17 19:45:50,661 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 77.57 2021-05-17 19:45:55,743 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 80.8 2021-05-17 19:46:00,838 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 83.93 2021-05-17 19:46:05,944 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 86.94 2021-05-17 19:46:11,038 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 90.19 2021-05-17 19:46:16,111 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 93.17 2021-05-17 19:46:21,211 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 96.21 2021-05-17 19:46:26,335 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 99.12 2021-05-17 19:46:31,408 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:46:36,477 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:46:46,568 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:46:46,720 | test | INFO | pool-13-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 229.101000071 sec 2021-05-17 19:46:46,773 | test | WARNING | Thread-31411 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:Rj7ukWQ7lcnepom45-7-569000 create failed keys from task: [u'test_collections-2314', u'test_collections-1108', u'test_collections-1647', u'test_collections-3120', u'test_collections-2378', u'test_collections-3831', u'test_collections-2128', u'test_collections-2824', u'test_collections-2647', u'test_collections-2284', u'test_collections-1098', u'test_collections-1177', u'test_collections-2466'] 2021-05-17 19:46:46,809 | test | WARNING | Thread-31511 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:6h88kg-4-632000:qNvBCM8RcyWHg-4-900000 create failed keys from task: [u'test_collections-745', u'test_collections-2314', u'test_collections-2833', u'test_collections-1605', u'test_collections-4395', u'test_collections-4577', u'test_collections-2378', u'test_collections-3831', u'test_collections-4205', u'test_collections-2284', u'test_collections-4024', u'test_collections-3652', u'test_collections-2466', u'test_collections-3137', u'test_collections-1795'] 2021-05-17 19:46:46,815 | test | WARNING | Thread-31512 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:6h88kg-4-632000:-%TfYp-4-899000 create failed keys from task: [u'test_collections-3326', u'test_collections-1159', u'test_collections-3879', u'test_collections-1135'] 2021-05-17 19:46:46,828 | test | WARNING | Thread-31588 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:TBp-4-414000 create failed keys from task: [u'test_collections-906', u'test_collections-4835', u'test_collections-3405', u'test_collections-3068', u'test_collections-1442', u'test_collections-896', u'test_collections-4425', u'test_collections-3595', u'test_collections-1330', u'test_collections-3377', u'test_collections-1553', u'test_collections-4642'] 2021-05-17 19:46:46,875 | test | WARNING | Thread-31513 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AhyhxxlLNRzR1D-5-523000 create failed keys from task: [u'test_collections-648', u'test_collections-745', u'test_collections-1501', u'test_collections-2833', u'test_collections-4395', u'test_collections-4577', u'test_collections-1562', u'test_collections-839', u'test_collections-1380', u'test_collections-1972', u'test_collections-3831', u'test_collections-4205', u'test_collections-1491', u'test_collections-4024', u'test_collections-1273', u'test_collections-3652', u'test_collections-1210', u'test_collections-2466', u'test_collections-3137'] 2021-05-17 19:46:46,875 | test | WARNING | Thread-31514 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ZjPloV771CuaxXfrYkV2x-5-329000:AuOR-5-523000 create failed keys from task: [u'test_collections-916', u'test_collections-516', u'test_collections-249', u'test_collections-886', u'test_collections-3469', u'test_collections-486'] 2021-05-17 19:46:46,923 | test | WARNING | Thread-31587 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:vrz9%Fc-VIpTAGYAKi-4-207000:HpfSPLpA7pBi-4-415000 create failed keys from task: [u'test_collections-745', u'test_collections-2314', u'test_collections-2833', u'test_collections-1605', u'test_collections-4395', u'test_collections-4577', u'test_collections-2378', u'test_collections-3831', u'test_collections-4205', u'test_collections-2284', u'test_collections-4024', u'test_collections-3652', u'test_collections-2466', u'test_collections-3137', u'test_collections-1795'] 2021-05-17 19:46:55,256 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:47:48,967 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 19:47:48,969 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 19:48:04,746 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 19:48:04,747 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 19:48:28,357 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 19:48:28,358 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 19:49:20,950 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1360001564 seconds to complete 2021-05-17 19:49:20,964 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 819.955s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 11 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_11 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_12 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_12', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 12, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 19:49:30,362 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:49:31,049 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:49:31,515 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3fd2f6 type=client open_count=1 channel=[id: 0x8b7294bc, 0.0.0.0/0.0.0.0:51316] timeout=300.0>) 2021-05-17 19:49:52,431 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3fd2f7 type=client open_count=1 channel=[id: 0xfea571d7, 0.0.0.0/0.0.0.0:54348] timeout=300.0>) 2021-05-17 19:49:54,451 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x3fd2f8 type=client open_count=1 channel=[id: 0xe21b018b, 0.0.0.0/0.0.0.0:34190] timeout=300.0>) 2021-05-17 19:49:56,470 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3fd2f9 type=client open_count=1 channel=[id: 0xf81192a8, 0.0.0.0/0.0.0.0:54047] timeout=300.0>) 2021-05-17 19:50:20,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3fd2fa type=client open_count=1 channel=[id: 0x8bf42fed, 0.0.0.0/0.0.0.0:41813] timeout=300.0>) 2021-05-17 19:50:22,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x3fd2fb type=client open_count=1 channel=[id: 0xdd1b64e0, 0.0.0.0/0.0.0.0:44075] timeout=300.0>) 2021-05-17 19:50:24,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3fd2fc type=client open_count=1 channel=[id: 0x2b336853, 0.0.0.0/0.0.0.0:46365] timeout=300.0>) 2021-05-17 19:50:48,678 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3fd2fd type=client open_count=1 channel=[id: 0x1f6e5e43, 0.0.0.0/0.0.0.0:56701] timeout=300.0>) 2021-05-17 19:50:50,697 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x3fd2fe type=client open_count=1 channel=[id: 0x8b4ed0ab, 0.0.0.0/0.0.0.0:51921] timeout=300.0>) 2021-05-17 19:50:52,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3fd2ff type=client open_count=1 channel=[id: 0x792712d1, 0.0.0.0/0.0.0.0:45449] timeout=300.0>) 2021-05-17 19:51:16,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3fd300 type=client open_count=1 channel=[id: 0x0f3c9c76, 0.0.0.0/0.0.0.0:38243] timeout=300.0>) 2021-05-17 19:51:18,641 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x3fd301 type=client open_count=1 channel=[id: 0x187ade52, 0.0.0.0/0.0.0.0:39834] timeout=300.0>) 2021-05-17 19:51:20,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3fd302 type=client open_count=1 channel=[id: 0x7578fe79, 0.0.0.0/0.0.0.0:43731] timeout=300.0>) 2021-05-17 19:51:44,971 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3fd303 type=client open_count=1 channel=[id: 0xc012ab62, 0.0.0.0/0.0.0.0:53392] timeout=300.0>) 2021-05-17 19:51:46,996 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x3fd304 type=client open_count=1 channel=[id: 0x8ec24a78, 0.0.0.0/0.0.0.0:49433] timeout=300.0>) 2021-05-17 19:51:49,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 19:52:06,361 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 19:52:20,305 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 19:52:20,326 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:52:20,365 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #12 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:52:30,690 | test | INFO | pool-14-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.57481417286 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:52:30,714 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:52:40,792 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:52:40,821 | test | INFO | pool-14-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.131000042 sec 2021-05-17 19:52:41,085 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 19:53:26,496 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 19:53:26,769 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 606174816 | 108954755 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 838860800 | 219612096 | 136 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 649560480 | 92011687 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 19:53:26,769 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 19:54:03,467 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:54:03,476 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:54:55,098 | test | INFO | pool-14-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......***...*...**........................................*..............................................*X | 101.927 | | 81.647 | ...............................................................................X | 81.647 | | 58.417 | ........................................................***........................*X | 83.039 | +--------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 19:54:56,680 | test | INFO | pool-14-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +----------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..........***.............................................*............................................*X | 10729.006 | | 9934.720 | ...........................................................................................X | 9934.720 | | 9145.115 | ....................................................................................**.......***...**X | 10277.090 | +----------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:54:56,740 | test | INFO | pool-14-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.....***...............................................................................................**X | 115.462 | | 106.696 | ...........................................................................................X | 106.696 | | 97.131 | ...................................................................................*X | 97.131 | | 32.154 | ..........................***......**X | 38.571 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 19:54:57,098 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.4783159439 | 6437847040 | 4086972416 | 2630230016 / 7511588864 | 104751 / 204897 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.7653714951 | 6437847040 | 4135104512 | 2618298368 / 7511588864 | 126000 / 255200 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.8973339957 | 6437847040 | 3595276288 | 3322277888 / 9530753024 | 126800 / 249200 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9484132927 | 6437847040 | 4176068608 | 2600734720 / 7511588864 | 104492 / 209766 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 19:55:09,045 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 19:56:21,390 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 158792368 | 231856178 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 838860800 | 242255008 | 136 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 541589840 | 405812683 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 19:56:21,391 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 19:56:29,680 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-17 19:56:29,680 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-17 19:56:33,482 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 19:56:33,680 | test | INFO | pool-14-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 16.2269810849 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 16.2244591847 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 25.4966666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 17.4213763104 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 19:56:33,707 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:56:38,802 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 19:56:42,786 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 19:56:43,903 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.46 2021-05-17 19:56:49,290 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 1.11 2021-05-17 19:56:50,927 | test | INFO | pool-14-thread-8 | [table_view:display:72] Keys failed in bucket1:VrlU-40-510000:5V3q5-bKw-40-650000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:50,927 | test | INFO | pool-14-thread-8 | [table_view:display:72] Keys failed in bucket1:VrlU-40-510000:5V3q5-bKw-40-650000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:50,934 | test | INFO | pool-14-thread-12 | [table_view:display:72] Keys failed in bucket1:dLikuVxxUA3cN-40-800000:YFGh%qF3gqhLxRZBwIf8-40-966000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:50,934 | test | INFO | pool-14-thread-12 | [table_view:display:72] Keys failed in bucket1:dLikuVxxUA3cN-40-800000:YFGh%qF3gqhLxRZBwIf8-40-966000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,056 | test | INFO | pool-14-thread-26 | [table_view:display:72] Keys failed in bucket1:ICEarbXxeu_aXHufHr-41-147000:wc5zI_I6-41-308000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,058 | test | INFO | pool-14-thread-26 | [table_view:display:72] Keys failed in bucket1:ICEarbXxeu_aXHufHr-41-147000:wc5zI_I6-41-308000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,063 | test | INFO | pool-14-thread-25 | [table_view:display:72] Keys failed in bucket1:dLikuVxxUA3cN-40-800000:5pEMaU0ce2Ow9-40-966000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,065 | test | INFO | pool-14-thread-25 | [table_view:display:72] Keys failed in bucket1:dLikuVxxUA3cN-40-800000:5pEMaU0ce2Ow9-40-966000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,068 | test | INFO | pool-14-thread-3 | [table_view:display:72] Keys failed in bucket1:VrlU-40-510000:MWYVQiB7eR-40-651000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:51,068 | test | INFO | pool-14-thread-3 | [table_view:display:72] Keys failed in bucket1:VrlU-40-510000:MWYVQiB7eR-40-651000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4228 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 19:56:52,328 | test | ERROR | pool-14-thread-9 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 19:56:52,329 | test | ERROR | pool-14-thread-9 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 19:56:52,467 | test | ERROR | pool-14-thread-9 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 19:56:52,469 | test | ERROR | pool-14-thread-9 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 19:56:52,469 | test | ERROR | pool-14-thread-9 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 19:56:54,671 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 2.03 2021-05-17 19:56:59,894 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 3.39 2021-05-17 19:57:05,233 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-05-17 19:57:10,622 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 6.72 2021-05-17 19:57:16,243 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 8.27 2021-05-17 19:57:19,638 | test | INFO | pool-14-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..*..................................................................................................**X | 3478.723 | | 27.610 | *.X | 43.705 | +--------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:57:21,112 | test | INFO | pool-14-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.*...................................................................................................*X | 10261.316 | | 9496.465 | ...........................................................................................*X | 9496.465 | | 9339.129 | ..........................................................................................X | 9339.129 | +----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 19:57:21,398 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 11.77 2021-05-17 19:57:22,000 | test | INFO | pool-14-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....*................................................................................................**X | 2918.295 | | 44.025 | *X | 55.876 | | 54.402 | X | 54.402 | +--------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 19:57:26,588 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 15.33 2021-05-17 19:57:31,713 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 19.34 2021-05-17 19:57:36,838 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 24.16 2021-05-17 19:57:41,936 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 29.01 2021-05-17 19:57:47,032 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:57:52,096 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:57:57,217 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 19:58:02,457 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 39.58 2021-05-17 19:58:07,565 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 48.75 2021-05-17 19:58:12,657 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 57.29 2021-05-17 19:58:17,740 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 63.91 2021-05-17 19:58:22,809 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:58:27,877 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 19:58:33,138 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 67.42 2021-05-17 19:58:38,242 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 72.11 2021-05-17 19:58:43,323 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 76.84 2021-05-17 19:58:48,414 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 81.63 2021-05-17 19:58:53,512 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 86.21 2021-05-17 19:58:58,648 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 91.0 2021-05-17 19:59:03,769 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 95.1 2021-05-17 19:59:08,857 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 99.17 2021-05-17 19:59:13,936 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:59:19,000 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 19:59:29,084 | test | INFO | pool-14-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 19:59:29,181 | test | INFO | pool-14-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 175.501999855 sec 2021-05-17 19:59:29,276 | test | WARNING | Thread-34432 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:VrlU-40-510000:MWYVQiB7eR-40-651000 create failed keys from task: [u'test_collections-4228'] 2021-05-17 19:59:29,296 | test | WARNING | Thread-34505 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:ICEarbXxeu_aXHufHr-41-147000:wc5zI_I6-41-308000 create failed keys from task: [u'test_collections-4228'] 2021-05-17 19:59:29,276 | test | WARNING | Thread-34429 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dLikuVxxUA3cN-40-800000:YFGh%qF3gqhLxRZBwIf8-40-966000 create failed keys from task: [u'test_collections-3665'] 2021-05-17 19:59:29,276 | test | WARNING | Thread-34431 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:VrlU-40-510000:5V3q5-bKw-40-650000 create failed keys from task: [u'test_collections-3665'] 2021-05-17 19:59:29,348 | test | WARNING | Thread-34430 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dLikuVxxUA3cN-40-800000:5pEMaU0ce2Ow9-40-966000 create failed keys from task: [u'test_collections-4228'] 2021-05-17 19:59:39,648 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 20:00:44,832 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 20:00:44,834 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 20:00:57,805 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 20:00:57,805 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 20:01:23,615 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 20:01:23,617 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 20:02:20,740 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1380000114 seconds to complete 2021-05-17 20:02:20,755 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 779.804s OK During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 12 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_12 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_13 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_13', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'num_items': '10000', 'bucket_spec': 'single_bucket.default_ephemeral', 'case_number': 13, 'bulk_api_crud': 'True'} test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 20:02:30,615 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #13 test_rebalance_cycles ========= 2021-05-17 20:02:31,345 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #13 test_rebalance_cycles ========= 2021-05-17 20:02:31,898 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x459bec type=client open_count=1 channel=[id: 0x93e18216, 0.0.0.0/0.0.0.0:58539] timeout=300.0>) 2021-05-17 20:02:52,714 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x459bed type=client open_count=1 channel=[id: 0x74dac24d, 0.0.0.0/0.0.0.0:38252] timeout=300.0>) 2021-05-17 20:02:54,734 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x459bee type=client open_count=1 channel=[id: 0xd05a8054, 0.0.0.0/0.0.0.0:39243] timeout=300.0>) 2021-05-17 20:02:56,752 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x459bef type=client open_count=1 channel=[id: 0xde965531, 0.0.0.0/0.0.0.0:37100] timeout=300.0>) 2021-05-17 20:03:20,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x459bf0 type=client open_count=1 channel=[id: 0x9b5fb47d, 0.0.0.0/0.0.0.0:36531] timeout=300.0>) 2021-05-17 20:03:22,895 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x459bf1 type=client open_count=1 channel=[id: 0xec96b6eb, 0.0.0.0/0.0.0.0:46398] timeout=300.0>) 2021-05-17 20:03:24,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x459bf2 type=client open_count=1 channel=[id: 0xbb116b53, 0.0.0.0/0.0.0.0:40659] timeout=300.0>) 2021-05-17 20:03:48,974 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x459bf3 type=client open_count=1 channel=[id: 0x6043e38e, 0.0.0.0/0.0.0.0:35608] timeout=300.0>) 2021-05-17 20:03:50,993 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x459bf4 type=client open_count=1 channel=[id: 0xdbff2a19, 0.0.0.0/0.0.0.0:54585] timeout=300.0>) 2021-05-17 20:03:53,013 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x459bf5 type=client open_count=1 channel=[id: 0x8dc7a2b7, 0.0.0.0/0.0.0.0:49741] timeout=300.0>) 2021-05-17 20:04:17,181 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x459bf6 type=client open_count=1 channel=[id: 0xc304d884, 0.0.0.0/0.0.0.0:44513] timeout=300.0>) 2021-05-17 20:04:19,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x459bf7 type=client open_count=1 channel=[id: 0x364b3eeb, 0.0.0.0/0.0.0.0:49413] timeout=300.0>) 2021-05-17 20:04:21,224 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x459bf8 type=client open_count=1 channel=[id: 0x55c5e2dc, 0.0.0.0/0.0.0.0:33876] timeout=300.0>) 2021-05-17 20:04:45,220 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x459bf9 type=client open_count=1 channel=[id: 0x93d867a5, 0.0.0.0/0.0.0.0:37533] timeout=300.0>) 2021-05-17 20:04:47,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x459bfa type=client open_count=1 channel=[id: 0x99b9b7af, 0.0.0.0/0.0.0.0:51253] timeout=300.0>) 2021-05-17 20:04:49,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 20:05:06,341 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 20:05:20,138 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 20:05:20,161 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #13 test_rebalance_cycles ========= 2021-05-17 20:05:20,198 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #13 test_rebalance_cycles ========= 2021-05-17 20:05:30,693 | test | INFO | pool-15-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 11.1733333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:05:30,719 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:05:40,796 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:05:40,823 | test | INFO | pool-15-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.131000042 sec 2021-05-17 20:05:41,141 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 20:05:41,371 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 20:05:41,505 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Ibxd8-41-190000 | ephemeral | 2 | none | 0 | 0 | 19167969280 | 0 | 0 | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 20:05:41,506 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 20:05:43,196 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:05:43,204 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:05:44,621 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 1.14174278285 | 6437847040 | 4699680768 | 1945923584 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 0.263324555848 | 6437847040 | 4741181440 | 1908293632 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 0.858654774187 | 6437847040 | 4209528832 | 2596798464 / 9530753024 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 0.445 | 6437847040 | 4783509504 | 1898790912 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 20:05:47,573 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 20:06:03,533 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Ibxd8-41-190000 | ephemeral | 2 | none | 0 | 10000 | 19167969280 | 95856016 | 136 | +-----------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 20:06:03,535 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #13 test_rebalance_cycles ========= 2021-05-17 20:06:08,680 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 0 2021-05-17 20:06:08,743 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:06:08,750 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:06:08,923 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:06:10,953 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:06:12,586 | test | INFO | pool-15-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 12.2106105305 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.8684978083 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 10.0241606265 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 11.5085251421 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:06:12,611 | test | INFO | pool-15-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:06:13,089 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:06:17,676 | test | INFO | pool-15-thread-17 | [task:check:355] Rebalance - status: running, progress: 3.07 2021-05-17 20:06:22,746 | test | INFO | pool-15-thread-17 | [task:check:355] Rebalance - status: running, progress: 47.31 2021-05-17 20:06:27,839 | test | INFO | pool-15-thread-17 | [task:check:355] Rebalance - status: running, progress: 91.36 2021-05-17 20:06:37,931 | test | INFO | pool-15-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:06:37,960 | test | INFO | pool-15-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 25.3740000725 sec 2021-05-17 20:06:38,006 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:06:38,012 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:06:38,058 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:06:38,140 | test | INFO | pool-15-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 42.1214296446 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 35.8066666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 33.3100011666 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 54.0875681261 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 30.3133333333 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:06:38,206 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:06:40,457 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:06:43,325 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 22.36 2021-05-17 20:06:48,426 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 52.46 2021-05-17 20:06:53,509 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 76.72 2021-05-17 20:06:58,579 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:07:03,641 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:07:13,737 | test | INFO | pool-15-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:07:14,355 | test | INFO | pool-15-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 36.2149999142 sec 2021-05-17 20:07:14,355 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 1 2021-05-17 20:07:14,401 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:07:14,405 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:07:14,437 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:07:16,473 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:07:18,104 | test | INFO | pool-15-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 35.1304956501 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 39.8423307055 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 51.9290702132 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:07:18,124 | test | INFO | pool-15-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:07:18,582 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:07:23,628 | test | INFO | pool-15-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.58 2021-05-17 20:07:28,726 | test | INFO | pool-15-thread-1 | [task:check:355] Rebalance - status: running, progress: 37.66 2021-05-17 20:07:33,826 | test | INFO | pool-15-thread-1 | [task:check:355] Rebalance - status: running, progress: 71.37 2021-05-17 20:07:43,927 | test | INFO | pool-15-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:07:43,970 | test | INFO | pool-15-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 25.8659999371 sec 2021-05-17 20:07:44,032 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:07:44,039 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:07:44,088 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:07:44,174 | test | INFO | pool-15-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 40.765224359 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 37.5797876773 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 32.6038767313 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 57.7845558945 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:07:44,226 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:07:46,221 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:07:49,368 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 23.83 2021-05-17 20:07:54,457 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 64.77 2021-05-17 20:07:59,625 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 93.91 2021-05-17 20:08:04,701 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:08:09,763 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:08:19,865 | test | INFO | pool-15-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:08:20,025 | test | INFO | pool-15-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 35.8509998322 sec 2021-05-17 20:08:20,025 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 2 2021-05-17 20:08:20,084 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:08:20,088 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:08:20,135 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:08:22,170 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:08:23,990 | test | INFO | pool-15-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 5.83490275162 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 5.10175169586 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 6.40811972934 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:08:24,010 | test | INFO | pool-15-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:08:24,286 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:08:29,082 | test | INFO | pool-15-thread-16 | [task:check:355] Rebalance - status: running, progress: 2.93 2021-05-17 20:08:34,167 | test | INFO | pool-15-thread-16 | [task:check:355] Rebalance - status: running, progress: 35.39 2021-05-17 20:08:39,252 | test | INFO | pool-15-thread-16 | [task:check:355] Rebalance - status: running, progress: 67.58 2021-05-17 20:08:49,348 | test | INFO | pool-15-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:08:49,384 | test | INFO | pool-15-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 25.3930001259 sec 2021-05-17 20:08:49,430 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:08:49,437 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:08:49,473 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:08:49,556 | test | INFO | pool-15-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 51.5390778887 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 49.0367308846 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 47.130504457 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 62.1618919054 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:08:49,598 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:08:51,582 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:08:54,700 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 24.3 2021-05-17 20:08:59,783 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 63.48 2021-05-17 20:09:04,865 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 91.21 2021-05-17 20:09:09,931 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:09:14,993 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:09:25,081 | test | INFO | pool-15-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:09:25,233 | test | INFO | pool-15-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 35.6760001183 sec 2021-05-17 20:09:25,233 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 3 2021-05-17 20:09:25,279 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:09:25,286 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:09:25,325 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:09:27,355 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:09:29,226 | test | INFO | pool-15-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 29.3226440881 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 30.6255104252 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 37.8133333333 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:09:29,247 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:09:29,454 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:09:34,342 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: running, progress: 3.2 2021-05-17 20:09:39,410 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: running, progress: 34.45 2021-05-17 20:09:44,487 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: running, progress: 63.05 2021-05-17 20:09:49,558 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: running, progress: 95.9 2021-05-17 20:09:59,645 | test | INFO | pool-15-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:09:59,676 | test | INFO | pool-15-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 30.4490001202 sec 2021-05-17 20:09:59,724 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:09:59,730 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:09:59,835 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 20:09:59,904 | test | INFO | pool-15-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 52.914118098 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 47.605 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 36.38 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 55.3986378205 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:09:59,937 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:10:01,969 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 20:10:05,072 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 23.71 2021-05-17 20:10:10,187 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 63.59 2021-05-17 20:10:15,267 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 90.51 2021-05-17 20:10:20,348 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:10:25,408 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:10:35,493 | test | INFO | pool-15-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:10:35,634 | test | INFO | pool-15-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 35.7309999466 sec 2021-05-17 20:10:36,553 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: Ibxd8-41-190000, Active Resident ratio(DGM): 100% 2021-05-17 20:10:36,555 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: Ibxd8-41-190000, Replica Resident ratio(DGM): 100% 2021-05-17 20:10:54,124 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.129999876 seconds to complete 2021-05-17 20:10:54,138 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 513.308s OK During the test, Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 13 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_13 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_14 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY,replicas=2,bucket_spec=single_bucket.default_ephemeral,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY' Test Input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_14', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY', 'num_items': '10000', 'bucket_spec': 'single_bucket.default_ephemeral', 'case_number': 14, 'bulk_api_crud': 'True'} test_Orchestrator_Node_failover (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 20:11:03,480 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #14 test_Orchestrator_Node_failover ========= 2021-05-17 20:11:04,177 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #14 test_Orchestrator_Node_failover ========= 2021-05-17 20:11:04,621 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b08a type=client open_count=1 channel=[id: 0xd4c6c0b3, 0.0.0.0/0.0.0.0:42201] timeout=300.0>) 2021-05-17 20:11:25,420 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b08b type=client open_count=1 channel=[id: 0x302d39ef, 0.0.0.0/0.0.0.0:42890] timeout=300.0>) 2021-05-17 20:11:27,438 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b08c type=client open_count=1 channel=[id: 0xefaa9acf, 0.0.0.0/0.0.0.0:35699] timeout=300.0>) 2021-05-17 20:11:29,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b08d type=client open_count=1 channel=[id: 0x86290c10, 0.0.0.0/0.0.0.0:33597] timeout=300.0>) 2021-05-17 20:11:53,362 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b08e type=client open_count=1 channel=[id: 0x0a43ee84, 0.0.0.0/0.0.0.0:40485] timeout=300.0>) 2021-05-17 20:11:55,381 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b08f type=client open_count=1 channel=[id: 0xcfdb5d13, 0.0.0.0/0.0.0.0:41036] timeout=300.0>) 2021-05-17 20:11:57,403 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b090 type=client open_count=1 channel=[id: 0xbc3411f0, 0.0.0.0/0.0.0.0:40889] timeout=300.0>) 2021-05-17 20:12:21,539 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b091 type=client open_count=1 channel=[id: 0xda3fc6d9, 0.0.0.0/0.0.0.0:57101] timeout=300.0>) 2021-05-17 20:12:23,571 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b092 type=client open_count=1 channel=[id: 0x30e0c5ff, 0.0.0.0/0.0.0.0:54150] timeout=300.0>) 2021-05-17 20:12:25,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b093 type=client open_count=1 channel=[id: 0xf5aa8a61, 0.0.0.0/0.0.0.0:51254] timeout=300.0>) 2021-05-17 20:12:49,964 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b094 type=client open_count=1 channel=[id: 0x620ec242, 0.0.0.0/0.0.0.0:57251] timeout=300.0>) 2021-05-17 20:12:51,983 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b095 type=client open_count=1 channel=[id: 0x6d31e172, 0.0.0.0/0.0.0.0:59441] timeout=300.0>) 2021-05-17 20:12:54,000 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b096 type=client open_count=1 channel=[id: 0x2d514aaf, 0.0.0.0/0.0.0.0:46736] timeout=300.0>) 2021-05-17 20:13:18,016 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b097 type=client open_count=1 channel=[id: 0x4da15f56, 0.0.0.0/0.0.0.0:58346] timeout=300.0>) 2021-05-17 20:13:20,035 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b098 type=client open_count=1 channel=[id: 0xab446c87, 0.0.0.0/0.0.0.0:60410] timeout=300.0>) 2021-05-17 20:13:22,052 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 20:13:39,559 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 20:13:53,315 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 20:13:53,341 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #14 test_Orchestrator_Node_failover ========= 2021-05-17 20:13:53,382 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #14 test_Orchestrator_Node_failover ========= 2021-05-17 20:14:09,109 | test | INFO | pool-16-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.156909876976 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 20:14:09,138 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:14:19,213 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:14:19,240 | test | INFO | pool-16-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1319999695 sec 2021-05-17 20:14:19,492 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 20:14:19,700 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 20:14:19,835 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | 9idriO6p-19-534000 | ephemeral | 2 | none | 0 | 0 | 19167969280 | 0 | 0 | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 20:14:19,835 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 20:14:21,601 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:14:21,608 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:14:23,107 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 13.3321110704 | 6437847040 | 4669079552 | 2059730944 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 4.55674261238 | 6437847040 | 4677656576 | 2022645760 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 7.3867607415 | 6437847040 | 4131684352 | 2729840640 / 9530753024 | 0 / 0 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 0.288328527858 | 6437847040 | 4778291200 | 1901785088 / 7511588864 | 0 / 0 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 20:14:26,006 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 20:14:42,286 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | 9idriO6p-19-534000 | ephemeral | 2 | none | 0 | 10000 | 19167969280 | 95866112 | 136 | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 20:14:42,289 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #14 test_Orchestrator_Node_failover ========= 2021-05-17 20:14:47,825 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 0 2021-05-17 20:14:57,003 | test | INFO | pool-16-thread-17 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.10199999809 seconds to complete 2021-05-17 20:14:57,007 | test | INFO | pool-16-thread-17 | [common_lib:sleep:22] Sleep 8.10199999809 seconds. Reason: Wait after rebalance complete 2021-05-17 20:15:05,257 | test | INFO | pool-16-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 2.73333333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 2.65986700665 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.676666666667 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 2.68004466741 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 20:15:05,282 | test | INFO | pool-16-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:15:10,349 | test | INFO | pool-16-thread-18 | [task:check:355] Rebalance - status: running, progress: 26.13 2021-05-17 20:15:15,417 | test | INFO | pool-16-thread-18 | [task:check:355] Rebalance - status: running, progress: 69.21 Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b199 type=client open_count=1 channel=[id: 0x27b04ec7, 0.0.0.0/0.0.0.0:48296] timeout=300.0>) 2021-05-17 20:15:20,459 | test | ERROR | pool-16-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-17 20:15:31,075 | test | INFO | pool-16-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:15:31,098 | test | ERROR | pool-16-thread-18 | [rest_client:_http_request:748] GET http://172.23.107.126:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.107.126:8091/pools/default with status 0: unknown pool 2021-05-17 20:15:31,160 | test | INFO | pool-16-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 25.9030001163 sec 2021-05-17 20:15:31,276 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.127 2021-05-17 20:15:31,276 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.126 2021-05-17 20:15:36,601 | test | INFO | pool-16-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 4.94825086249 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 4.00993316778 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.35104326378 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:15:36,628 | test | INFO | pool-16-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:15:41,710 | test | INFO | pool-16-thread-19 | [task:check:355] Rebalance - status: running, progress: 5.39 2021-05-17 20:15:46,808 | test | INFO | pool-16-thread-19 | [task:check:355] Rebalance - status: running, progress: 40.31 2021-05-17 20:15:51,904 | test | INFO | pool-16-thread-19 | [task:check:355] Rebalance - status: running, progress: 81.59 2021-05-17 20:16:01,990 | test | INFO | pool-16-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:16:02,019 | test | INFO | pool-16-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 25.4179999828 sec 2021-05-17 20:16:02,019 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 1 2021-05-17 20:16:11,165 | test | INFO | pool-16-thread-20 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.08500003815 seconds to complete 2021-05-17 20:16:11,171 | test | INFO | pool-16-thread-20 | [common_lib:sleep:22] Sleep 8.08500003815 seconds. Reason: Wait after rebalance complete 2021-05-17 20:16:19,397 | test | INFO | pool-16-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 4.3780414639 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 7.23845397423 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 11.2470415681 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 5.13158114031 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:16:19,420 | test | INFO | pool-16-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:16:24,476 | test | INFO | pool-16-thread-21 | [task:check:355] Rebalance - status: running, progress: 20.9 2021-05-17 20:16:29,539 | test | INFO | pool-16-thread-21 | [task:check:355] Rebalance - status: running, progress: 58.21 2021-05-17 20:16:34,601 | test | INFO | pool-16-thread-21 | [task:check:355] Rebalance - status: running, progress: 95.61 2021-05-17 20:16:44,686 | test | INFO | pool-16-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:16:44,775 | test | INFO | pool-16-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 25.378000021 sec 2021-05-17 20:16:44,868 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.126 2021-05-17 20:16:44,868 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.127 2021-05-17 20:16:48,740 | test | INFO | pool-16-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | unknown | None | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 22.3703728395 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 24.2162630623 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 24.0362660622 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:16:48,763 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:16:53,812 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:16:58,878 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: running, progress: 12.72 2021-05-17 20:17:04,029 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: running, progress: 46.48 2021-05-17 20:17:09,134 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: running, progress: 85.52 2021-05-17 20:17:19,240 | test | INFO | pool-16-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:17:19,269 | test | INFO | pool-16-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 30.5299999714 sec 2021-05-17 20:17:19,270 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 2 2021-05-17 20:17:28,467 | test | INFO | pool-16-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.06799983978 seconds to complete 2021-05-17 20:17:28,473 | test | INFO | pool-16-thread-23 | [common_lib:sleep:22] Sleep 8.06799983978 seconds. Reason: Wait after rebalance complete 2021-05-17 20:17:36,680 | test | INFO | pool-16-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 2.83838063968 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 2.65820042331 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.545802479408 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 2.73652984017 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 20:17:36,701 | test | INFO | pool-16-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:17:41,766 | test | INFO | pool-16-thread-24 | [task:check:355] Rebalance - status: running, progress: 28.04 2021-05-17 20:17:46,819 | test | INFO | pool-16-thread-24 | [task:check:355] Rebalance - status: running, progress: 71.75 Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b19a type=client open_count=1 channel=[id: 0xac7d6d3c, 0.0.0.0/0.0.0.0:59792] timeout=300.0>) 2021-05-17 20:17:51,861 | test | ERROR | pool-16-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-17 20:18:01,809 | test | INFO | pool-16-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:18:01,828 | test | ERROR | pool-16-thread-24 | [rest_client:_http_request:748] GET http://172.23.107.126:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.107.126:8091/pools/default with status 0: unknown pool 2021-05-17 20:18:01,890 | test | INFO | pool-16-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 25.2100000381 sec 2021-05-17 20:18:01,980 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.127 2021-05-17 20:18:01,980 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.126 2021-05-17 20:18:06,740 | test | INFO | pool-16-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 5.18316056131 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 3.82993616773 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.08984850252 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:18:06,763 | test | INFO | pool-16-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:18:11,842 | test | INFO | pool-16-thread-25 | [task:check:355] Rebalance - status: running, progress: 2.97 2021-05-17 20:18:16,914 | test | INFO | pool-16-thread-25 | [task:check:355] Rebalance - status: running, progress: 37.06 2021-05-17 20:18:21,996 | test | INFO | pool-16-thread-25 | [task:check:355] Rebalance - status: running, progress: 76.21 2021-05-17 20:18:32,085 | test | INFO | pool-16-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:18:32,130 | test | INFO | pool-16-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 25.3900001049 sec 2021-05-17 20:18:32,131 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 3 2021-05-17 20:18:41,305 | test | INFO | pool-16-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.07800006866 seconds to complete 2021-05-17 20:18:41,309 | test | INFO | pool-16-thread-26 | [common_lib:sleep:22] Sleep 8.07800006866 seconds. Reason: Wait after rebalance complete 2021-05-17 20:18:49,539 | test | INFO | pool-16-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 4.42666666667 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 6.11833333333 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.05008416807 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 5.6620150898 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:18:49,563 | test | INFO | pool-16-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:18:54,634 | test | INFO | pool-16-thread-27 | [task:check:355] Rebalance - status: running, progress: 27.25 2021-05-17 20:18:59,696 | test | INFO | pool-16-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.5 2021-05-17 20:19:09,776 | test | INFO | pool-16-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:19:09,865 | test | INFO | pool-16-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 20.3259999752 sec 2021-05-17 20:19:09,934 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 20:19:10,984 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 20:19:12,036 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 20:19:13,085 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 20:19:14,150 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.126 2021-05-17 20:19:14,150 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.127 2021-05-17 20:19:17,977 | test | INFO | pool-16-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 12.4983333333 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.7097048383 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 12.3447942534 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:19:18,002 | test | INFO | pool-16-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:19:23,055 | test | INFO | pool-16-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:19:28,128 | test | INFO | pool-16-thread-28 | [task:check:355] Rebalance - status: running, progress: 19.3 2021-05-17 20:19:33,207 | test | INFO | pool-16-thread-28 | [task:check:355] Rebalance - status: running, progress: 56.37 2021-05-17 20:19:43,313 | test | INFO | pool-16-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:19:43,351 | test | INFO | pool-16-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 25.375 sec 2021-05-17 20:19:44,323 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: 9idriO6p-19-534000, Active Resident ratio(DGM): 100% 2021-05-17 20:19:44,325 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: 9idriO6p-19-534000, Replica Resident ratio(DGM): 100% 2021-05-17 20:20:01,374 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 20:20:01,390 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 547.270s OK During the test, Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 14 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_14 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_15 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_15', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 15, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 20:20:10,888 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #15 test_data_load_collections_with_rebalance_in ========= 2021-05-17 20:20:11,625 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #15 test_data_load_collections_with_rebalance_in ========= 2021-05-17 20:20:12,081 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b19b type=client open_count=1 channel=[id: 0xd59eb9dc, 0.0.0.0/0.0.0.0:49645] timeout=300.0>) 2021-05-17 20:20:32,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b19c type=client open_count=1 channel=[id: 0x3a24d297, 0.0.0.0/0.0.0.0:36583] timeout=300.0>) 2021-05-17 20:20:34,940 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x45b19d type=client open_count=1 channel=[id: 0x43d954a8, 0.0.0.0/0.0.0.0:46533] timeout=300.0>) 2021-05-17 20:20:36,959 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b19e type=client open_count=1 channel=[id: 0x9983fdc4, 0.0.0.0/0.0.0.0:56706] timeout=300.0>) 2021-05-17 20:21:01,286 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b19f type=client open_count=1 channel=[id: 0x1d03a838, 0.0.0.0/0.0.0.0:35018] timeout=300.0>) 2021-05-17 20:21:03,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b1a0 type=client open_count=1 channel=[id: 0x4b22d7b8, 0.0.0.0/0.0.0.0:60548] timeout=300.0>) 2021-05-17 20:21:05,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x45b1a1 type=client open_count=1 channel=[id: 0x53b644fe, 0.0.0.0/0.0.0.0:48316] timeout=300.0>) 2021-05-17 20:21:07,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b1a2 type=client open_count=1 channel=[id: 0x8608e37d, 0.0.0.0/0.0.0.0:40099] timeout=300.0>) 2021-05-17 20:21:31,005 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b1a3 type=client open_count=1 channel=[id: 0xea47f52a, 0.0.0.0/0.0.0.0:54252] timeout=300.0>) 2021-05-17 20:21:33,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x45b1a4 type=client open_count=1 channel=[id: 0x1b15145c, 0.0.0.0/0.0.0.0:53486] timeout=300.0>) 2021-05-17 20:21:35,042 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b1a5 type=client open_count=1 channel=[id: 0x5580c5db, 0.0.0.0/0.0.0.0:57848] timeout=300.0>) 2021-05-17 20:21:59,210 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b1a6 type=client open_count=1 channel=[id: 0xeafe17fa, 0.0.0.0/0.0.0.0:50985] timeout=300.0>) 2021-05-17 20:22:01,227 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x45b1a7 type=client open_count=1 channel=[id: 0x679446b2, 0.0.0.0/0.0.0.0:39233] timeout=300.0>) 2021-05-17 20:22:03,246 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b1a8 type=client open_count=1 channel=[id: 0xa04ed807, 0.0.0.0/0.0.0.0:41446] timeout=300.0>) 2021-05-17 20:22:27,684 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b1a9 type=client open_count=1 channel=[id: 0x49cf6465, 0.0.0.0/0.0.0.0:40915] timeout=300.0>) 2021-05-17 20:22:29,703 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x45b1aa type=client open_count=1 channel=[id: 0xae271b06, 0.0.0.0/0.0.0.0:38173] timeout=300.0>) 2021-05-17 20:22:31,720 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 20:22:49,076 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 20:23:02,960 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 20:23:02,983 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #15 test_data_load_collections_with_rebalance_in ========= 2021-05-17 20:23:03,025 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #15 test_data_load_collections_with_rebalance_in ========= 2021-05-17 20:23:10,029 | test | INFO | pool-17-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 15.2340950381 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:23:10,055 | test | INFO | pool-17-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:23:20,131 | test | INFO | pool-17-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:23:20,158 | test | INFO | pool-17-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1309998035 sec 2021-05-17 20:23:20,413 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 20:24:07,720 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 20:24:07,974 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 575990472 | 55889797 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 673655952 | 72081126 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 1090965016 | 38840263 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-17 20:24:07,976 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 20:24:35,522 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:24:35,532 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:46:07,717 | test | INFO | pool-17-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****......******....*.............*****............................................................................*****X | 50.905 | | 1.566 | ..*****X | 1.566 | | 1.066 | .X | 1.066 | | 0.000 | *************************************..******.....................................................*****X | 28.469 | | 22.668 | ...........................................X | 22.668 | | 10.498 | ...................****X | 10.498 | | 3.410 | .....X | 3.410 | | 0.000 | *****........******......*****X | 7.503 | | 6.502 | ...........X | 6.502 | | 4.501 | .......******...............*****X | 12.007 | | 3.903 | ......******................................*****X | 19.904 | | 9.202 | .................****X | 9.202 | | 0.000 | ******************.....***********X | 3.001 | | 3.001 | ....*****X | 3.001 | | 0.000 | ******************************..*......****X | 4.502 | | 4.202 | .......X | 4.202 | | 4.201 | .......****X | 4.201 | | 3.702 | ......X | 3.702 | | 3.299 | .....*****.*.****X | 4.202 | | 0.300 | ******.*..*****X | 2.001 | | 0.999 | *****.*****X | 1.500 | | 0.000 | *************************..*****X | 1.500 | | 0.000 | ******...*..****X | 3.000 | | 2.001 | ..X | 2.001 | | 0.000 | ******************......*..............****X | 10.502 | | 8.004 | ..............X | 8.004 | | 4.001 | ......******..X | 5.001 | | 5.000 | ........****X | 5.000 | | 0.000 | ******.*.****X | 1.500 | | 0.501 | X | 0.501 | | 0.000 | *****************...*..****X | 3.000 | | 2.501 | ...X | 2.501 | | 1.500 | .*****.*..*****X | 2.999 | | 0.000 | ************************........*****X | 4.500 | | 3.000 | ....X | 3.000 | | 0.000 | *****.......*....*****X | 6.001 | | 1.500 | .*****X | 1.500 | | 1.500 | .*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ************************.*.***********..*.****X | 3.000 | | 3.000 | ....*****X | 3.000 | | 2.000 | ..X | 2.000 | | 1.500 | .****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *****.....*****X | 3.000 | | 3.000 | ....*****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | *****..........*****X | 5.500 | | 0.500 | ******..******...*****X | 3.000 | | 1.500 | .*****X | 1.500 | | 0.000 | *************************........*****X | 4.500 | | 1.500 | .X | 1.500 | | 0.000 | ******************************.....*****X | 3.000 | | 0.000 | ************************************************.....*****X | 3.001 | | 0.000 | ********************..*****..*.*****X | 3.000 | | 0.000 | *******.*****.....*****X | 3.500 | | 0.000 | ************..*****X | 1.500 | | 0.100 | ******........*****X | 4.398 | | 2.899 | ....X | 2.899 | | 0.000 | ******************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 20:46:08,516 | test | INFO | pool-17-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****........................*..................................................****X | 408.274 | | 375.991 | ...................................................................X | 375.991 | | 322.562 | .........................................................****X | 322.562 | | 312.929 | ........................................................******................................****X | 492.402 | | 434.131 | ..............................................................................*.....******.*****X | 469.268 | | 443.065 | ...............................................................................******..******...*..*****....*****X | 501.366 | | 397.194 | .......................................................................X | 397.194 | | 334.535 | ............................................................****X | 334.535 | | 306.694 | .......................................................X | 306.694 | | 293.283 | ....................................................*****...................*....****X | 420.734 | | 397.248 | .......................................................................*..*****.*****X | 411.720 | | 377.220 | ...................................................................******.*.....*****X | 405.579 | | 373.161 | ...................................................................******..*****X | 386.474 | | 356.528 | ................................................................******........*****............*****X | 467.114 | | 351.918 | ...............................................................******.........*****X | 401.910 | | 379.291 | ....................................................................******..******....*****X | 415.368 | | 373.668 | ...................................................................******........******....*.*****X | 444.201 | | 405.188 | .........................................................................X | 405.188 | | 373.019 | ...................................................................*****.......*****X | 415.741 | | 393.247 | ......................................................................*****X | 393.247 | | 373.883 | ...................................................................X | 373.883 | | 356.669 | ................................................................****X | 356.669 | | 336.396 | ............................................................X | 336.396 | | 307.098 | .......................................................*****..X | 321.151 | | 320.797 | .........................................................*****............*****X | 383.583 | | 366.961 | ..................................................................X | 366.961 | | 360.464 | ................................................................*****...*.****X | 381.210 | | 374.231 | ...................................................................******.....*..*****X | 413.453 | | 391.758 | ......................................................................******.....*****X | 417.931 | | 373.952 | ...................................................................******..........X | 429.732 | | 422.736 | ............................................................................****X | 422.736 | | 412.979 | ..........................................................................*.****X | 416.521 | | 410.142 | .........................................................................*...******........*****X | 470.433 | | 450.182 | .................................................................................X | 450.182 | | 407.030 | .........................................................................****X | 407.030 | | 379.148 | ....................................................................*.....*****.X | 410.865 | | 369.377 | ..................................................................*****X | 369.377 | | 346.563 | ..............................................................*****X | 346.563 | | 330.538 | ...........................................................******........******.X | 379.832 | | 355.018 | ...............................................................*****....X | 376.239 | | 363.549 | .................................................................*****.............*****X | 434.111 | | 380.999 | ....................................................................******.....*******..****X | 419.841 | | 376.371 | ...................................................................X | 376.371 | | 348.609 | ..............................................................*****....X | 367.164 | | 367.159 | ..................................................................*****......*.****X | 407.419 | | 366.826 | ..................................................................*..*****....X | 400.097 | | 395.808 | .......................................................................****X | 395.808 | | 352.741 | ...............................................................X | 352.741 | | 319.981 | .........................................................*****.........*....****X | 393.529 | | 381.654 | ....................................................................X | 381.654 | | 349.072 | ..............................................................****X | 349.072 | | 323.949 | ..........................................................X | 323.949 | | 304.041 | ......................................................*****.*..******........*****X | 364.318 | | 328.613 | ...........................................................X | 328.613 | | 321.321 | .........................................................*****.******.....*.*****.....*.****X | 391.812 | | 388.010 | .....................................................................*......*****..X | 430.434 | | 401.473 | ........................................................................****X | 401.473 | | 326.272 | ..........................................................X | 326.272 | | 314.040 | ........................................................*****.........*...*****....*X | 401.994 | | 394.693 | .......................................................................****X | 394.693 | | 387.829 | .....................................................................******.*****X | 390.372 | | 388.586 | ......................................................................X | 388.586 | | 380.320 | ....................................................................****X | 380.320 | | 325.324 | ..........................................................*****......*.....****X | 383.489 | | 377.378 | ...................................................................X | 377.378 | | 359.775 | ................................................................*****.X | 364.027 | | 357.391 | ................................................................*****....*...******.****X | 404.586 | | 395.075 | .......................................................................*..*****X | 406.676 | | 366.776 | ..................................................................******.....*****X | 394.906 | | 354.713 | ...............................................................******..******..........*****X | 421.158 | | 360.812 | ................................................................******....*****X | 382.388 | | 381.912 | ....................................................................******...*...****X | 410.714 | | 410.614 | ..........................................................................*****X | 410.614 | | 380.711 | ....................................................................******X | 380.711 | | 374.349 | ...................................................................*****..X | 385.287 | | 366.062 | .................................................................****X | 366.062 | | 349.612 | ..............................................................******.....*.......****X | 414.731 | | 390.281 | ......................................................................X | 390.281 | | 354.565 | ...............................................................****X | 354.565 | | 353.768 | ...............................................................*****X | 353.768 | | 325.270 | ..........................................................*******...******............*****X | 406.972 | | 400.523 | ........................................................................******....X | 422.086 | | 416.173 | ...........................................................................*****X | 416.173 | | 338.967 | ............................................................******............*****X | 403.470 | | 385.900 | .....................................................................*****X | 385.900 | | 371.135 | ..................................................................******.........*****X | 420.296 | | 413.982 | ..........................................................................X | 413.982 | | 385.498 | .....................................................................******.*****X | 389.600 | | 355.089 | ...............................................................******.........*****X | 403.651 | | 399.707 | ........................................................................******.*****X | 406.372 | | 387.410 | .....................................................................X | 387.410 | | 372.340 | ...................................................................****X | 372.340 | | 351.415 | ...............................................................******.......******.............*.........*****.......*****X | 547.135 | | 387.878 | .....................................................................X | 387.878 | +---------+-----------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 20:46:09,878 | test | INFO | pool-17-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.....................******...******............................................................................*****X | 42.404 | | 16.853 | ......................................*****X | 16.853 | | 0.000 | *************************************............................*................*****.....................................*................****X | 41.154 | | 14.198 | ................................X | 14.198 | | 1.500 | ..****X | 1.500 | | 0.502 | X | 0.502 | | 0.000 | ************..........*****X | 4.302 | | 2.498 | ....X | 2.498 | | 1.700 | ...*****.............*****X | 7.504 | | 7.203 | ...............X | 7.203 | | 6.704 | ..............*****X | 6.704 | | 0.799 | *****........*...............****X | 10.502 | | 0.000 | *******************....*...****X | 3.001 | | 0.000 | ************************************************..*.....*****.*..*****.......*....****X | 9.003 | | 4.004 | ........X | 4.004 | | 1.500 | ..*****.....*.........*****X | 7.500 | | 3.001 | ......*****X | 3.001 | | 3.000 | ......*****X | 3.000 | | 0.000 | ******...*..****X | 2.501 | | 1.501 | ..X | 1.501 | | 0.500 | ****X | 0.500 | | 0.000 | ******.*..*****X | 1.500 | | 0.000 | ******..*.****X | 1.500 | | 0.000 | ******....*...****X | 3.000 | | 2.001 | ...X | 2.001 | | 0.000 | ******..*.****X | 1.500 | | 0.000 | ******.*..****X | 1.500 | | 1.001 | .X | 1.001 | | 0.501 | X | 0.501 | | 0.000 | ************************************...******..*..*****................*........****X | 13.502 | | 4.504 | .........*X | 4.504 | | 0.000 | ****************...*.......****X | 4.500 | | 3.000 | ......X | 3.000 | | 0.000 | ******.......*****X | 3.001 | | 2.000 | ...X | 2.000 | | 1.500 | ..****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ******************************************....*...****X | 3.000 | | 2.000 | ...X | 2.000 | | 1.500 | ..****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ***********....................*****X | 8.501 | | 4.500 | .........X | 4.500 | | 2.000 | ...****X | 2.000 | | 0.500 | X | 0.500 | | 0.000 | *****..*.****X | 1.500 | | 1.500 | ..X | 1.500 | | 1.500 | ..****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *******************************************.......*****X | 3.000 | | 0.000 | ******************************...*****X | 1.500 | | 1.000 | .X | 1.000 | | 0.000 | *****************.*..****X | 1.500 | | 0.000 | ************....*...****X | 3.000 | | 2.000 | ...X | 2.000 | | 1.500 | ..*****..................*****X | 9.002 | | 0.000 | ******.*..****X | 1.500 | | 1.000 | .X | 1.000 | | 0.000 | ******.................*****X | 7.500 | | 3.000 | ......*****X | 3.000 | | 1.500 | ..******X | 1.500 | | 0.400 | ******......*****X | 2.600 | | 0.000 | ************...*****X | 1.500 | | 0.000 | ********************************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 20:46:10,144 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 73.4382292153 | 6437847040 | 4089692160 | 2689163264 / 7511588864 | 168632 / 336916 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 77.0029500492 | 6437847040 | 4043964416 | 2690375680 / 7511588864 | 168580 / 336867 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 78.4920251329 | 6437847040 | 3726045184 | 3143499776 / 9530753024 | 168267 / 337227 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 20:46:19,213 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 20:47:12,825 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 162424648 | 259784623 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 159037008 | 273029551 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 437062984 | 467509671 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 20:47:12,825 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #15 test_data_load_collections_with_rebalance_in ========= 2021-05-17 20:47:19,141 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in 2021-05-17 20:47:19,142 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-17 20:47:19,197 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 20:47:27,434 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 20:47:29,686 | test | INFO | pool-17-thread-26 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 18.6986449774 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 17.659411353 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.084430519 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 20:47:30,007 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:47:35,535 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 20:47:40,846 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.84 2021-05-17 20:47:46,005 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 5.73 2021-05-17 20:47:51,142 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 8.6 2021-05-17 20:47:56,256 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 11.27 2021-05-17 20:48:01,365 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.39 2021-05-17 20:48:06,492 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 17.35 2021-05-17 20:48:11,598 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.67 2021-05-17 20:48:16,780 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 22.89 2021-05-17 20:48:21,927 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 25.95 2021-05-17 20:48:27,049 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 28.61 2021-05-17 20:48:28,473 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:28,474 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-360 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:32,161 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.53 2021-05-17 20:48:34,075 | test | INFO | pool-17-thread-7 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:v2vT0clj9hgsotMMmtMm-25-727000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:34,075 | test | INFO | pool-17-thread-7 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:v2vT0clj9hgsotMMmtMm-25-727000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:36,881 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-455 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:36,881 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-455 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:37,252 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.31 2021-05-17 20:48:42,351 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.62 2021-05-17 20:48:44,444 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:44,444 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in bucket1:nNe-25-557000:G7d4CBN-25-727000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-613 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-783 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:48:47,562 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 37.49 2021-05-17 20:48:52,733 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 41.31 2021-05-17 20:48:57,927 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 43.33 2021-05-17 20:49:03,105 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 47.21 2021-05-17 20:49:06,272 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:06,273 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1161 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:08,311 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-17 20:49:08,747 | test | INFO | pool-17-thread-9 | [table_view:display:72] Keys failed in bucket2:hj_sNQAinGpyUsiJ4-24-402000:Ij6O6JRwZXYBQ7-24-670000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:08,749 | test | INFO | pool-17-thread-9 | [table_view:display:72] Keys failed in bucket2:hj_sNQAinGpyUsiJ4-24-402000:Ij6O6JRwZXYBQ7-24-670000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1009 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1199 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:13,466 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 52.46 2021-05-17 20:49:15,513 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:15,515 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1265 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:18,595 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 54.81 2021-05-17 20:49:23,700 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 57.0 2021-05-17 20:49:24,670 | test | INFO | pool-17-thread-9 | [table_view:display:72] Keys failed in bucket2:hj_sNQAinGpyUsiJ4-24-402000:Ij6O6JRwZXYBQ7-24-670000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:24,671 | test | INFO | pool-17-thread-9 | [table_view:display:72] Keys failed in bucket2:hj_sNQAinGpyUsiJ4-24-402000:Ij6O6JRwZXYBQ7-24-670000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1547 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1541 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:28,842 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 59.71 2021-05-17 20:49:29,075 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1635 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:29,075 | test | INFO | pool-17-thread-14 | [table_view:display:72] Keys failed in bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1677 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1635 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:33,976 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 62.12 2021-05-17 20:49:39,115 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 64.56 2021-05-17 20:49:44,240 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.22 2021-05-17 20:49:50,005 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.74 2021-05-17 20:49:55,213 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.52 2021-05-17 20:49:55,404 | test | INFO | pool-17-thread-10 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-721 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:49:55,937 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-329 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:00,367 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 72.96 2021-05-17 20:50:01,263 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:tVyhwHIJ-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:02,032 | test | INFO | pool-17-thread-3 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:V8ws-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-936 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-925 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-973 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:05,523 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 76.31 2021-05-17 20:50:07,865 | test | INFO | pool-17-thread-13 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:V8ws-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-417 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-415 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-487 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-422 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-411 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:07,936 | test | INFO | pool-17-thread-4 | [table_view:display:72] Keys failed in default:_default:sc8UqjHQ-DD_anvPjb-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-535 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-587 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-563 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-581 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:08,168 | test | INFO | pool-17-thread-6 | [table_view:display:72] Keys failed in default:scope2:70QUPYeC-20-491000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-936 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-925 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-973 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:08,174 | test | INFO | pool-17-thread-18 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:9fik-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-936 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-925 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-973 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:09,328 | test | INFO | pool-17-thread-15 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:sYi-eqrQT0Wyw9q3cK-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-639 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-535 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-677 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-532 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-587 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-563 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-585 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-541 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-691 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-581 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:10,637 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 78.94 2021-05-17 20:50:11,257 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in default:scope2:0Ryz00ZN-20-493000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-925 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-912 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-960 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:13,155 | test | INFO | pool-17-thread-10 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-968 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-912 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:14,295 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:14,296 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:15,799 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 81.25 2021-05-17 20:50:17,934 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-416 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-435 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-454 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-411 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-486 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-494 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-471 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:21,038 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 83.8 2021-05-17 20:50:21,950 | test | INFO | pool-17-thread-5 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:s-26-826000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-692 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:21,950 | test | INFO | pool-17-thread-5 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:s-26-826000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-702 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-653 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-692 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:23,841 | test | INFO | pool-17-thread-20 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:H25Z-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-908 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-935 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-947 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-933 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-940 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:23,845 | test | INFO | pool-17-thread-28 | [table_view:display:72] Keys failed in default:GHeDehOfDfNIhhuq3xG-26-998000:FkaFmZI3wJVQyku1-27-107000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-803 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-993 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:23,846 | test | INFO | pool-17-thread-28 | [table_view:display:72] Keys failed in default:GHeDehOfDfNIhhuq3xG-26-998000:FkaFmZI3wJVQyku1-27-107000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-803 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-993 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:24,197 | test | INFO | pool-17-thread-23 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:4TTP0feH1zU-20-494000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:24,197 | test | INFO | pool-17-thread-19 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:9KUVrNWAfhQ-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:26,180 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 85.79 2021-05-17 20:50:28,910 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:tVyhwHIJ-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:29,252 | test | INFO | pool-17-thread-1 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:LVynD8Pg2-20-496000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-629 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-509 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-515 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-615 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-624 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-536 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-666 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-685 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-575 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-550 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-592 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:29,681 | test | INFO | pool-17-thread-4 | [table_view:display:72] Keys failed in default:_default:sc8UqjHQ-DD_anvPjb-20-486000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-717 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-737 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-823 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-862 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:31,292 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 87.27 2021-05-17 20:50:36,201 | test | INFO | pool-17-thread-15 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:sYi-eqrQT0Wyw9q3cK-20-495000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-727 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-725 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-833 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-740 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-894 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-873 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:36,401 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 88.59 2021-05-17 20:50:40,869 | test | INFO | pool-17-thread-19 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:9KUVrNWAfhQ-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-904 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-970 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:40,869 | test | INFO | pool-17-thread-11 | [table_view:display:72] Keys failed in default:_default:q3ouubpvA%9o7sY-20-485000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-904 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-970 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:41,299 | test | INFO | pool-17-thread-22 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:d-4DdAQ7lfliHI_o1D-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-507 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-546 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-514 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-569 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-503 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-665 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-699 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-600 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-551 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-595 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-695 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-640 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-690 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:41,506 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 90.55 2021-05-17 20:50:41,802 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-17 20:50:43,055 | test | INFO | pool-17-thread-4 | [table_view:display:72] Keys failed in default:_default:sc8UqjHQ-DD_anvPjb-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-956 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:43,058 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:tVyhwHIJ-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-913 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-941 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:43,737 | test | INFO | pool-17-thread-1 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:LVynD8Pg2-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-879 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-700 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-883 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-790 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:44,821 | test | INFO | pool-17-thread-23 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:4TTP0feH1zU-20-494000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-974 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:46,630 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 92.09 2021-05-17 20:50:50,783 | test | INFO | pool-17-thread-15 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:sYi-eqrQT0Wyw9q3cK-20-495000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-972 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:51,770 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 93.4 2021-05-17 20:50:52,332 | test | INFO | pool-17-thread-11 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:c%3qd-G4ljj-20-493000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-564 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-584 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:53,042 | test | INFO | pool-17-thread-19 | [table_view:display:72] Keys failed in default:_default:t5A-38pVM9es-20-487000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-627 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-521 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-584 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:54,918 | test | INFO | pool-17-thread-6 | [table_view:display:72] Keys failed in default:_default:Z-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-638 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-617 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-562 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-683 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:55,323 | test | INFO | pool-17-thread-1 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:LVynD8Pg2-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-972 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:55,904 | test | INFO | pool-17-thread-22 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:d-4DdAQ7lfliHI_o1D-20-496000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-828 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-713 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-722 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-756 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-841 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-787 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:50:56,947 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 94.77 2021-05-17 20:50:58,381 | test | INFO | pool-17-thread-2 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:8XJJm4KnTXeBUAGx_%0W-20-495000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-617 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-646 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-652 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-664 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-683 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:01,177 | test | INFO | pool-17-thread-23 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:v-20-494000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-646 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-621 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-654 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-664 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:01,815 | test | INFO | pool-17-thread-8 | [table_view:display:72] Keys failed in default:_default:WLNrtllAoP1G_hotCSL7-20-485000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-27 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-32 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-80 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-134 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:02,068 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 96.18 2021-05-17 20:51:02,496 | test | INFO | pool-17-thread-20 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:H25Z-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-27 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-80 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:05,165 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in default:GHeDehOfDfNIhhuq3xG-26-998000:6zGe8KLSVxRuZ1RUMf-27-107000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:05,167 | test | INFO | pool-17-thread-25 | [table_view:display:72] Keys failed in default:GHeDehOfDfNIhhuq3xG-26-998000:6zGe8KLSVxRuZ1RUMf-27-107000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:06,589 | test | INFO | pool-17-thread-10 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:9KUVrNWAfhQ-20-495000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-85 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-116 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-136 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-80 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:06,875 | test | INFO | pool-17-thread-6 | [table_view:display:72] Keys failed in default:_default:Z-20-486000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-817 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-865 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-870 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:07,220 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 97.29 2021-05-17 20:51:09,266 | test | INFO | pool-17-thread-11 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:c%3qd-G4ljj-20-493000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-817 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-835 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-865 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-870 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:09,867 | test | INFO | pool-17-thread-3 | [table_view:display:72] Keys failed in default:_default:IJblsejK5QpPAs--20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-598 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:10,648 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in default:_default:7GTuteWyee6NP4NOeU--20-485000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-597 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-661 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-571 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-572 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:10,756 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1550 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:10,757 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1550 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:12,313 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 98.09 2021-05-17 20:51:12,773 | test | INFO | pool-17-thread-22 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:d-4DdAQ7lfliHI_o1D-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-948 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-911 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:13,635 | test | INFO | pool-17-thread-23 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:v-20-494000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-852 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-881 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:14,282 | test | INFO | pool-17-thread-13 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:B3YxEMAqV-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-558 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-534 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-501 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:16,688 | test | INFO | pool-17-thread-19 | [table_view:display:72] Keys failed in default:_default:t5A-38pVM9es-20-487000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-768 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-774 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-881 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:17,405 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 99.35 2021-05-17 20:51:20,167 | test | INFO | pool-17-thread-18 | [table_view:display:72] Keys failed in default:_default:tNs6-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:23,137 | test | INFO | pool-17-thread-11 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:c%3qd-G4ljj-20-493000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-900 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:23,859 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 20:51:25,448 | test | INFO | pool-17-thread-6 | [table_view:display:72] Keys failed in default:_default:Z-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:25,667 | test | INFO | pool-17-thread-17 | [table_view:display:72] Keys failed in default:_default:35vr8RG_%QG-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:25,997 | test | INFO | pool-17-thread-20 | [table_view:display:72] Keys failed in default:pGTVIe8i-20-493000:H25Z-20-496000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-236 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-273 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:26,000 | test | INFO | pool-17-thread-8 | [table_view:display:72] Keys failed in default:_default:WLNrtllAoP1G_hotCSL7-20-485000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-227 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-362 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:26,914 | test | INFO | pool-17-thread-3 | [table_view:display:72] Keys failed in default:_default:IJblsejK5QpPAs--20-486000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-741 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:28,523 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1651 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:28,525 | test | INFO | pool-17-thread-27 | [table_view:display:72] Keys failed in default:KQM-%P-26-692000:50FpW-27-271000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1651 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:30,471 | test | INFO | pool-17-thread-21 | [table_view:display:72] Keys failed in default:_default:7GTuteWyee6NP4NOeU--20-485000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:30,470 | test | INFO | pool-17-thread-18 | [table_view:display:72] Keys failed in default:_default:tNs6-20-486000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 20:51:33,993 | test | INFO | pool-17-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 20:51:34,042 | test | INFO | pool-17-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 244.355999947 sec 2021-05-17 20:56:10,779 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 20:56:10,779 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 20:56:10,783 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 20:56:10,783 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 20:56:10,785 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 20:56:10,786 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 6 docs from 500 to 980 2021-05-17 20:56:10,786 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 20:56:10,786 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 20:56:10,789 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 8 docs from 500 to 980 2021-05-17 20:56:10,789 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 20:56:10,789 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 9 docs from 0 to 500 2021-05-17 20:56:10,789 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-17 20:56:10,792 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 20:56:10,792 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 20:56:10,792 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 13 docs from 0 to 500 2021-05-17 20:56:10,793 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 34 docs from 500 to 980 2021-05-17 20:56:10,793 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 20 docs from 500 to 980 2021-05-17 20:56:10,795 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 25 docs from 500 to 980 2021-05-17 20:56:10,796 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 19 docs from 500 to 980 2021-05-17 20:56:10,796 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 38 docs from 500 to 980 2021-05-17 20:56:10,796 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 32 docs from 500 to 980 2021-05-17 20:56:10,799 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 29 docs from 500 to 980 2021-05-17 20:56:10,802 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 20:56:10,802 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 17 docs from 500 to 980 2021-05-17 20:56:10,802 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-17 20:56:10,805 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 20:56:10,806 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 16 docs from 0 to 500 2021-05-17 20:56:10,806 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 12 docs from 500 to 980 2021-05-17 20:56:42,706 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 20:56:42,707 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 7 docs from 0 to 500 2021-05-17 20:56:42,707 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 15 docs from 0 to 500 2021-05-17 20:56:42,709 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 8 docs from 500 to 980 2021-05-17 20:56:42,710 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 8 docs from 500 to 980 2021-05-17 20:56:42,710 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 20:56:42,711 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 3 docs from 500 to 980 2021-05-17 20:56:42,711 | test | ERROR | pool-17-thread-30 | [task:call:3809] Failed to load 3 docs from 500 to 980 2021-05-17 21:01:20,359 | test | INFO | pool-17-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.......***X | 8.300 | | 0.300 | ****.............***X | 16.093 | | 14.997 | ...........*.....***.................***X | 40.295 | | 37.011 | ...............................****...*....***X | 46.093 | | 40.987 | ..................................***X | 40.987 | | 12.202 | .........*...***.......***..............*.......X | 47.764 | | 47.764 | ........................................**.......***........*....****....X | 74.735 | | 74.216 | ...............................................................***.....*..........**X | 91.453 | | 91.344 | ..............................................................................*......***X | 99.141 | | 89.853 | ............................................................................***X | 90.072 | | 89.438 | ............................................................................****.......................X | 115.561 | | 106.976 | ...........................................................................................**X | 106.976 | | 69.545 | ...........................................................X | 69.545 | | 53.470 | .............................................***X | 53.470 | | 44.682 | .....................................*****......**********......X | 58.472 | | 57.779 | ................................................*****.X | 57.980 | | 56.682 | ................................................***X | 56.682 | | 50.281 | ..........................................*.....*****.......X | 64.379 | | 63.282 | .....................................................****X | 63.282 | | 54.881 | ..............................................*.***X | 56.081 | | 55.879 | ...............................................*****X | 55.882 | | 51.081 | ...........................................X | 51.081 | | 48.085 | ........................................*****...........****X | 61.181 | | 58.281 | .................................................*..****..X | 63.388 | | 59.891 | ..................................................*****...****X | 62.490 | | 58.689 | .................................................*..****X | 60.683 | | 52.975 | ............................................*.*****......X | 60.195 | | 56.996 | ................................................****.*.****X | 59.988 | | 10.787 | ........X | 10.787 | | 1.100 | *****..*****...........................*.......*****..........................*......****.*..****X | 83.196 | | 77.076 | .................................................................*****.........*.....*****X | 93.588 | | 89.589 | ............................................................................*****........*****X | 99.067 | | 81.000 | .....................................................................*****......................*..***X | 108.964 | | 103.567 | ........................................................................................X | 103.567 | | 81.908 | .....................................................................****X | 81.908 | | 64.089 | ......................................................X | 64.089 | | 60.487 | ...................................................*********X | 60.992 | | 60.991 | ...................................................X | 60.991 | | 59.692 | ..................................................****X | 59.692 | | 1.600 | *****.*.****X | 3.899 | | 1.599 | *X | 1.599 | | 0.500 | ****..*.****X | 4.100 | | 1.699 | X | 1.699 | | 0.700 | *****.*.***X | 2.479 | | 1.500 | X | 1.500 | | 0.800 | *****.*..****X | 3.499 | | 2.928 | .X | 2.928 | | 2.128 | *****.****X | 2.599 | | 1.299 | X | 1.299 | | 0.800 | *****.******..****X | 3.796 | | 2.995 | .X | 2.995 | | 2.295 | ****X | 2.295 | | 0.898 | X | 0.898 | | 0.798 | ****.**.***X | 2.498 | | 2.398 | .X | 2.398 | | 2.098 | *****.*.*****.****X | 4.703 | | 3.107 | .X | 3.107 | | 1.600 | X | 1.600 | | 0.901 | ****..****X | 3.102 | | 1.496 | X | 1.496 | | 0.400 | X | 0.400 | | 0.000 | ****..*.****X | 3.703 | | 2.603 | .****X | 2.603 | | 1.000 | X | 1.000 | | 0.600 | ****.*.****X | 2.501 | | 2.401 | .X | 2.401 | | 2.101 | ***X | 2.101 | | 1.101 | X | 1.101 | | 1.101 | *****....****X | 5.004 | | 4.404 | ..X | 4.404 | | 0.200 | X | 0.200 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:01:21,490 | test | INFO | pool-17-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................***X | 1268.643 | | 30.812 | .****.....***X | 95.257 | | 37.899 | .****............................***X | 383.151 | | 106.759 | .......****.*.***X | 134.084 | | 56.276 | ...****....................***X | 313.010 | | 126.574 | ........***.............................***X | 483.141 | | 216.704 | ................***X | 216.704 | | 156.212 | ...........***X | 156.212 | | 119.213 | ........*.*******X | 138.447 | | 74.601 | ....****...........*..****....***X | 287.014 | | 264.570 | ...................X | 264.570 | | 263.854 | ...................***.........*..........................***X | 707.173 | | 584.466 | .............................................X | 584.466 | | 367.810 | ...........................***X | 367.810 | | 308.054 | .......................X | 308.054 | | 283.911 | .....................***X | 283.911 | | 226.745 | ................X | 226.745 | | 223.839 | ................***X | 223.839 | | 217.723 | ................X | 217.723 | | 206.913 | ...............*****..******..........*X | 355.926 | | 355.926 | ...........................***X | 366.611 | | 345.972 | ..........................X | 345.972 | | 316.927 | .......................****.....*...****X | 408.372 | | 306.462 | .......................X | 306.462 | | 249.529 | ..................***X | 249.529 | | 242.534 | ..................X | 242.534 | | 228.119 | ................*X | 228.119 | | 228.119 | ................**X | 228.119 | | 214.524 | ...............X | 214.524 | | 210.130 | ...............****...*****.....*...*...****...*.****X | 433.423 | | 344.609 | ..........................X | 344.609 | | 266.228 | ...................****X | 266.228 | | 254.514 | ...................****X | 254.514 | | 251.314 | ..................*.*****.................*****..*..*****.X | 535.316 | | 530.575 | ........................................**X | 530.575 | | 530.575 | ........................................*X | 530.575 | | 523.451 | ........................................*****X | 523.451 | | 353.017 | ..........................*****.X | 364.162 | | 355.583 | ...........................****X | 355.583 | | 307.714 | .......................X | 307.714 | | 306.608 | .......................**X | 306.608 | | 306.608 | .......................X | 306.608 | | 304.826 | .......................X | 304.826 | | 301.158 | ......................***X | 301.158 | | 296.472 | ......................X | 296.472 | | 289.658 | .....................X | 289.658 | | 274.570 | ....................****..*.****X | 309.694 | | 300.606 | ......................*****X | 300.606 | | 297.302 | ......................*****........*...****X | 432.127 | | 213.846 | ...............*****.*.X | 238.863 | | 214.283 | ...............****...*..*****........***********......*..*****.*.****........*...*..X | 664.295 | | 664.295 | ...................................................**X | 664.295 | | 546.933 | ..........................................X | 546.933 | | 534.925 | .........................................***X | 534.925 | | 534.925 | .........................................*.X | 555.013 | | 481.380 | ....................................****X | 481.380 | | 435.353 | .................................X | 435.353 | | 433.244 | .................................****.*****X | 446.945 | | 432.340 | .................................******.......*.****.*.****X | 569.728 | | 515.525 | .......................................*X | 515.525 | | 495.631 | ......................................****.*.****...X | 566.289 | | 550.801 | ..........................................X | 550.801 | | 550.224 | ..........................................***X | 550.224 | | 519.278 | .......................................*...****X | 550.246 | | 493.032 | .....................................X | 493.032 | | 477.635 | ....................................****..X | 496.552 | | 487.795 | .....................................****X | 490.085 | | 480.386 | ....................................X | 480.386 | | 465.013 | ...................................****.*....****X | 530.338 | | 525.027 | ........................................X | 525.027 | | 434.700 | .................................*****..*.*****....*.***X | 533.558 | | 454.088 | ..................................*X | 454.088 | | 450.168 | ..................................****...........*.****X | 606.505 | | 605.835 | ..............................................****X | 605.835 | | 542.947 | .........................................X | 542.947 | | 517.688 | .......................................****..........*...****X | 680.093 | | 642.231 | .................................................****X | 642.231 | | 547.937 | ..........................................*X | 547.937 | | 527.563 | ........................................X | 527.563 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 21:01:21,953 | test | INFO | pool-17-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******X | 0.400 | | 0.000 | *******X | 1.397 | | 0.599 | ***...........*....****...........*..***........****.*.***X | 54.347 | | 46.585 | ................................**X | 46.585 | | 42.879 | .............................X | 42.879 | | 32.670 | ......................***......................................*..........***.X | 100.859 | | 100.161 | ......................................................................**X | 100.161 | | 80.273 | ........................................................****........*..**X | 94.818 | | 90.725 | ...............................................................*..***X | 92.660 | | 90.830 | ...............................................................X | 90.830 | | 81.757 | .........................................................**X | 81.757 | | 67.262 | ...............................................****....*..***...X | 79.958 | | 76.671 | .....................................................****......................*........***.........X | 130.750 | | 128.765 | ..........................................................................................***X | 128.765 | | 106.456 | ...........................................................................X | 106.456 | | 85.565 | ............................................................****X | 85.565 | | 70.774 | .................................................****X | 70.774 | | 67.472 | ...............................................*****.....................*.......*****....*..........****X | 126.643 | | 106.276 | ..........................................................................******......................******X | 136.751 | | 131.858 | .............................................................................................****...X | 136.254 | | 130.063 | ...........................................................................................***X | 130.063 | | 99.466 | ......................................................................******................*****.............X | 139.978 | | 130.381 | ............................................................................................****X | 130.381 | | 115.279 | .................................................................................****X | 115.279 | | 103.176 | ........................................................................*X | 103.176 | | 100.972 | .......................................................................****.................X | 124.676 | | 118.262 | ...................................................................................***X | 118.262 | | 112.057 | ...............................................................................*......****X | 121.191 | | 28.168 | ...................X | 28.168 | | 4.800 | ..****X | 4.800 | | 0.500 | *****X | 1.100 | | 0.900 | *****.*****X | 2.399 | | 1.299 | X | 1.299 | | 0.200 | *****.*****..*****X | 4.698 | | 0.700 | *****X | 0.800 | | 0.700 | *****.*****.****X | 2.800 | | 1.499 | X | 1.499 | | 0.400 | *****..................*****....*........................****X | 64.484 | | 58.194 | ........................................******...*****...........*..*****......*.........****X | 101.590 | | 86.488 | ............................................................X | 86.488 | | 85.087 | ...........................................................***X | 85.087 | | 63.287 | ............................................X | 63.287 | | 55.290 | ......................................X | 55.290 | | 43.799 | ..............................***X | 43.799 | | 35.699 | ........................X | 35.699 | | 4.899 | ..****X | 4.899 | | 3.119 | .*X | 3.119 | | 2.419 | ***X | 2.419 | | 1.100 | X | 1.100 | | 0.300 | ****..*.*..***X | 7.699 | | 5.299 | ..X | 5.299 | | 1.628 | *****.*..****X | 5.699 | | 1.799 | *****..****X | 5.594 | | 4.797 | ..X | 4.797 | | 4.597 | ..****X | 4.597 | | 4.595 | ..X | 4.595 | | 3.895 | .****X | 3.895 | | 3.197 | .X | 3.197 | | 2.297 | ******.****X | 3.098 | | 1.998 | ******.*****..*****X | 6.106 | | 1.499 | *****.****X | 3.802 | | 1.897 | X | 1.897 | | 1.000 | X | 1.000 | | 0.400 | ****....****X | 6.902 | | 6.302 | ...X | 6.302 | | 2.302 | ****X | 2.302 | | 1.902 | ******.****X | 4.102 | | 3.302 | .X | 3.302 | | 3.202 | .****X | 3.202 | | 2.802 | .X | 2.802 | | 2.602 | *****.X | 3.702 | | 3.702 | .***X | 3.702 | | 3.502 | .X | 3.502 | | 0.100 | *X | 0.100 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:01:22,026 | test | WARNING | Thread-37109 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:FBAYWs-23-470000:XudDC_vTjPbpn6_IJft-23-753000 create failed keys from task: [u'test_collections-1677', u'test_collections-1635', u'test_collections-1161', u'test_collections-1370', u'test_collections-1265'] 2021-05-17 21:01:22,040 | test | WARNING | Thread-37153 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:v-20-494000 update failed keys from task: [u'test_collections-726', u'test_collections-527', u'test_collections-524', u'test_collections-646', u'test_collections-522', u'test_collections-621', u'test_collections-567', u'test_collections-664', u'test_collections-642', u'test_collections-881', u'test_collections-654', u'test_collections-852'] 2021-05-17 21:01:22,042 | test | WARNING | Thread-37157 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:H25Z-20-496000 update failed keys from task: [u'test_collections-908', u'test_collections-926', u'test_collections-935', u'test_collections-947', u'test_collections-933', u'test_collections-940'] 2021-05-17 21:01:22,042 | test | WARNING | Thread-37155 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:sYi-eqrQT0Wyw9q3cK-20-495000 update failed keys from task: [u'test_collections-727', u'test_collections-526', u'test_collections-725', u'test_collections-967', u'test_collections-568', u'test_collections-641', u'test_collections-520', u'test_collections-563', u'test_collections-937', u'test_collections-535', u'test_collections-931', u'test_collections-657', u'test_collections-578', u'test_collections-850', u'test_collections-894', u'test_collections-532', u'test_collections-972', u'test_collections-691', u'test_collections-705', u'test_collections-548', u'test_collections-867', u'test_collections-744', u'test_collections-822', u'test_collections-740', u'test_collections-587', u'test_collections-585', u'test_collections-541', u'test_collections-581', u'test_collections-517', u'test_collections-639', u'test_collections-837', u'test_collections-516', u'test_collections-876', u'test_collections-833', u'test_collections-677', u'test_collections-754', u'test_collections-872', u'test_collections-873'] 2021-05-17 21:01:22,042 | test | WARNING | Thread-37158 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:9KUVrNWAfhQ-20-495000 update failed keys from task: [u'test_collections-904', u'test_collections-821', u'test_collections-866', u'test_collections-822', u'test_collections-785', u'test_collections-862', u'test_collections-717', u'test_collections-715', u'test_collections-913', u'test_collections-979', u'test_collections-812', u'test_collections-813', u'test_collections-898', u'test_collections-854', u'test_collections-910', u'test_collections-874', u'test_collections-752', u'test_collections-771', u'test_collections-772', u'test_collections-970'] 2021-05-17 21:01:22,042 | test | WARNING | Thread-37160 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:LVynD8Pg2-20-496000 update failed keys from task: [u'test_collections-849', u'test_collections-648', u'test_collections-769', u'test_collections-967', u'test_collections-887', u'test_collections-687', u'test_collections-685', u'test_collections-883', u'test_collections-880', u'test_collections-937', u'test_collections-539', u'test_collections-815', u'test_collections-977', u'test_collections-615', u'test_collections-934', u'test_collections-656', u'test_collections-536', u'test_collections-575', u'test_collections-972', u'test_collections-629', u'test_collections-509', u'test_collections-700', u'test_collections-624', u'test_collections-666', u'test_collections-543', u'test_collections-515', u'test_collections-879', u'test_collections-711', u'test_collections-754', u'test_collections-550', u'test_collections-592', u'test_collections-790'] 2021-05-17 21:01:22,043 | test | WARNING | Thread-37161 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:4TTP0feH1zU-20-494000 update failed keys from task: [u'test_collections-821', u'test_collections-866', u'test_collections-822', u'test_collections-785', u'test_collections-862', u'test_collections-939', u'test_collections-717', u'test_collections-715', u'test_collections-812', u'test_collections-813', u'test_collections-898', u'test_collections-854', u'test_collections-910', u'test_collections-874', u'test_collections-952', u'test_collections-974', u'test_collections-752', u'test_collections-771', u'test_collections-772'] 2021-05-17 21:01:22,045 | test | WARNING | Thread-37156 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:tVyhwHIJ-20-496000 update failed keys from task: [u'test_collections-939', u'test_collections-979', u'test_collections-812', u'test_collections-813', u'test_collections-898', u'test_collections-854', u'test_collections-578', u'test_collections-771', u'test_collections-772', u'test_collections-548', u'test_collections-821', u'test_collections-866', u'test_collections-822', u'test_collections-941', u'test_collections-785', u'test_collections-862', u'test_collections-585', u'test_collections-917', u'test_collections-717', u'test_collections-715', u'test_collections-913', u'test_collections-516', u'test_collections-910', u'test_collections-874', u'test_collections-752'] 2021-05-17 21:01:22,051 | test | WARNING | Thread-37176 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:V8ws-20-495000 update failed keys from task: [u'test_collections-906', u'test_collections-936', u'test_collections-925', u'test_collections-922', u'test_collections-978', u'test_collections-942', u'test_collections-943', u'test_collections-973'] 2021-05-17 21:01:22,051 | test | WARNING | Thread-37172 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:c%3qd-G4ljj-20-493000 update failed keys from task: [u'test_collections-909', u'test_collections-627', u'test_collections-900', u'test_collections-865', u'test_collections-646', u'test_collections-767', u'test_collections-564', u'test_collections-521', u'test_collections-584', u'test_collections-562', u'test_collections-818', u'test_collections-718', u'test_collections-817', u'test_collections-878', u'test_collections-835', u'test_collections-870', u'test_collections-651'] 2021-05-17 21:01:22,063 | test | WARNING | Thread-37189 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:d-4DdAQ7lfliHI_o1D-20-496000 update failed keys from task: [u'test_collections-722', u'test_collections-602', u'test_collections-569', u'test_collections-921', u'test_collections-841', u'test_collections-600', u'test_collections-640', u'test_collections-560', u'test_collections-699', u'test_collections-695', u'test_collections-690', u'test_collections-948', u'test_collections-828', u'test_collections-507', u'test_collections-669', u'test_collections-546', u'test_collections-503', u'test_collections-665', u'test_collections-787', u'test_collections-820', u'test_collections-713', u'test_collections-911', u'test_collections-635', u'test_collections-514', u'test_collections-756', u'test_collections-954', u'test_collections-675', u'test_collections-551', u'test_collections-595'] 2021-05-17 21:01:22,107 | test | WARNING | Thread-37349 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:nNe-25-557000:v2vT0clj9hgsotMMmtMm-25-727000 create failed keys from task: [u'test_collections-205', u'test_collections-395'] 2021-05-17 21:01:22,148 | test | WARNING | Thread-37348 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:nNe-25-557000:G7d4CBN-25-727000 create failed keys from task: [u'test_collections-613', u'test_collections-455', u'test_collections-783', u'test_collections-360'] 2021-05-17 21:01:22,151 | test | WARNING | Thread-37112 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:hj_sNQAinGpyUsiJ4-24-402000:Ij6O6JRwZXYBQ7-24-670000 create failed keys from task: [u'test_collections-1547', u'test_collections-1009', u'test_collections-1199', u'test_collections-1541'] 2021-05-17 21:01:22,805 | test | WARNING | Thread-37157 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:H25Z-20-496000 delete failed keys from task: [u'test_collections-308', u'test_collections-327', u'test_collections-126', u'test_collections-80', u'test_collections-362', u'test_collections-185', u'test_collections-2', u'test_collections-27', u'test_collections-85', u'test_collections-236', u'test_collections-116', u'test_collections-298', u'test_collections-153', u'test_collections-130', u'test_collections-273'] 2021-05-17 21:01:22,892 | test | WARNING | Thread-37176 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:V8ws-20-495000 delete failed keys from task: [u'test_collections-417', u'test_collections-428', u'test_collections-415', u'test_collections-446', u'test_collections-487', u'test_collections-422', u'test_collections-411', u'test_collections-486', u'test_collections-471'] 2021-05-17 21:01:22,892 | test | WARNING | Thread-37232 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:GHeDehOfDfNIhhuq3xG-26-998000:FkaFmZI3wJVQyku1-27-107000 create failed keys from task: [u'test_collections-803', u'test_collections-993'] 2021-05-17 21:01:22,971 | test | WARNING | Thread-37229 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KQM-%P-26-692000:s-26-826000 create failed keys from task: [u'test_collections-702', u'test_collections-653', u'test_collections-692'] 2021-05-17 21:01:22,980 | test | WARNING | Thread-37175 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:9fik-20-496000 update failed keys from task: [u'test_collections-906', u'test_collections-907', u'test_collections-925', u'test_collections-922', u'test_collections-942', u'test_collections-920', u'test_collections-943', u'test_collections-936', u'test_collections-958', u'test_collections-914', u'test_collections-978', u'test_collections-973', u'test_collections-951', u'test_collections-971'] 2021-05-17 21:01:22,999 | test | WARNING | Thread-37299 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:70QUPYeC-20-491000 update failed keys from task: [u'test_collections-906', u'test_collections-925', u'test_collections-922', u'test_collections-942', u'test_collections-943', u'test_collections-936', u'test_collections-958', u'test_collections-914', u'test_collections-978', u'test_collections-973', u'test_collections-971'] 2021-05-17 21:01:23,013 | test | WARNING | Thread-37182 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:8XJJm4KnTXeBUAGx_%0W-20-495000 update failed keys from task: [u'test_collections-524', u'test_collections-646', u'test_collections-522', u'test_collections-621', u'test_collections-567', u'test_collections-664', u'test_collections-642', u'test_collections-683', u'test_collections-617', u'test_collections-577', u'test_collections-652'] 2021-05-17 21:01:23,244 | test | WARNING | Thread-37193 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:sc8UqjHQ-DD_anvPjb-20-486000 update failed keys from task: [u'test_collections-526', u'test_collections-641', u'test_collections-563', u'test_collections-939', u'test_collections-737', u'test_collections-979', u'test_collections-535', u'test_collections-657', u'test_collections-578', u'test_collections-532', u'test_collections-772', u'test_collections-691', u'test_collections-829', u'test_collections-548', u'test_collections-823', u'test_collections-867', u'test_collections-866', u'test_collections-822', u'test_collections-941', u'test_collections-587', u'test_collections-785', u'test_collections-862', u'test_collections-585', u'test_collections-581', u'test_collections-917', u'test_collections-717', u'test_collections-517', u'test_collections-715', u'test_collections-913', u'test_collections-516', u'test_collections-956', u'test_collections-910', u'test_collections-752', u'test_collections-794'] 2021-05-17 21:01:23,244 | test | WARNING | Thread-37192 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:WLNrtllAoP1G_hotCSL7-20-485000 delete failed keys from task: [u'test_collections-32', u'test_collections-308', u'test_collections-227', u'test_collections-327', u'test_collections-126', u'test_collections-80', u'test_collections-362', u'test_collections-185', u'test_collections-2', u'test_collections-27', u'test_collections-85', u'test_collections-116', u'test_collections-298', u'test_collections-134', u'test_collections-153', u'test_collections-130'] 2021-05-17 21:01:23,365 | test | WARNING | Thread-37158 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:9KUVrNWAfhQ-20-495000 delete failed keys from task: [u'test_collections-2', u'test_collections-85', u'test_collections-116', u'test_collections-136', u'test_collections-80', u'test_collections-130', u'test_collections-185'] 2021-05-17 21:01:23,443 | test | WARNING | Thread-37276 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:0Ryz00ZN-20-493000 update failed keys from task: [u'test_collections-906', u'test_collections-925', u'test_collections-922', u'test_collections-920', u'test_collections-943', u'test_collections-960', u'test_collections-914', u'test_collections-912', u'test_collections-951', u'test_collections-971', u'test_collections-950'] 2021-05-17 21:01:23,443 | test | WARNING | Thread-37191 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:IJblsejK5QpPAs--20-486000 update failed keys from task: [u'test_collections-704', u'test_collections-757', u'test_collections-644', u'test_collections-688', u'test_collections-598', u'test_collections-741', u'test_collections-661', u'test_collections-574'] 2021-05-17 21:01:23,476 | test | WARNING | Thread-37221 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Z-20-486000 update failed keys from task: [u'test_collections-909', u'test_collections-865', u'test_collections-646', u'test_collections-767', u'test_collections-522', u'test_collections-562', u'test_collections-683', u'test_collections-818', u'test_collections-718', u'test_collections-817', u'test_collections-638', u'test_collections-617', u'test_collections-878', u'test_collections-835', u'test_collections-577', u'test_collections-652', u'test_collections-870', u'test_collections-651'] 2021-05-17 21:01:23,651 | test | WARNING | Thread-37214 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7GTuteWyee6NP4NOeU--20-485000 update failed keys from task: [u'test_collections-834', u'test_collections-688', u'test_collections-534', u'test_collections-630', u'test_collections-597', u'test_collections-661', u'test_collections-571', u'test_collections-572'] 2021-05-17 21:01:23,789 | test | WARNING | Thread-37185 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:pGTVIe8i-20-493000:B3YxEMAqV-20-496000 update failed keys from task: [u'test_collections-626', u'test_collections-558', u'test_collections-544', u'test_collections-534', u'test_collections-501'] 2021-05-17 21:01:23,793 | test | WARNING | Thread-37230 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:KQM-%P-26-692000:50FpW-27-271000 create failed keys from task: [u'test_collections-838', u'test_collections-1550', u'test_collections-1651'] 2021-05-17 21:01:23,875 | test | WARNING | Thread-37203 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:35vr8RG_%QG-20-486000 update failed keys from task: [u'test_collections-679', u'test_collections-694', u'test_collections-580'] 2021-05-17 21:01:23,881 | test | WARNING | Thread-37194 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:tNs6-20-486000 update failed keys from task: [u'test_collections-834', u'test_collections-544', u'test_collections-580'] 2021-05-17 21:01:23,976 | test | WARNING | Thread-37212 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-906', u'test_collections-805', u'test_collections-968', u'test_collections-920', u'test_collections-943', u'test_collections-721', u'test_collections-957', u'test_collections-914', u'test_collections-912', u'test_collections-951', u'test_collections-950'] 2021-05-17 21:01:23,983 | test | WARNING | Thread-37200 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:q3ouubpvA%9o7sY-20-485000 update failed keys from task: [u'test_collections-904', u'test_collections-913', u'test_collections-979', u'test_collections-910', u'test_collections-970'] 2021-05-17 21:01:24,030 | test | WARNING | Thread-37233 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:GHeDehOfDfNIhhuq3xG-26-998000:6zGe8KLSVxRuZ1RUMf-27-107000 create failed keys from task: [u'test_collections-27'] 2021-05-17 21:01:24,046 | test | WARNING | Thread-37224 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:t5A-38pVM9es-20-487000 update failed keys from task: [u'test_collections-627', u'test_collections-726', u'test_collections-768', u'test_collections-646', u'test_collections-521', u'test_collections-763', u'test_collections-584', u'test_collections-562', u'test_collections-881', u'test_collections-774', u'test_collections-651'] 2021-05-17 21:01:24,503 | test | WARNING | Thread-37212 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default delete failed keys from task: [u'test_collections-329', u'test_collections-428', u'test_collections-446', u'test_collections-486', u'test_collections-416', u'test_collections-435', u'test_collections-454', u'test_collections-411', u'test_collections-431', u'test_collections-494', u'test_collections-492', u'test_collections-350', u'test_collections-471'] 2021-05-17 21:01:37,742 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 21:03:01,105 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 95% 2021-05-17 21:03:01,107 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 95% 2021-05-17 21:03:21,161 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 21:03:21,163 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 21:03:49,693 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 21:03:49,694 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 21:04:44,194 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1290001869 seconds to complete 2021-05-17 21:04:44,210 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2682.838s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 15 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_15 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_16 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_16', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 16, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 21:04:53,799 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #16 test_data_load_collections_with_rebalance_out ========= 2021-05-17 21:04:54,470 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #16 test_data_load_collections_with_rebalance_out ========= 2021-05-17 21:04:54,897 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b8c4f type=client open_count=1 channel=[id: 0xdcd48a59, 0.0.0.0/0.0.0.0:58272] timeout=300.0>) 2021-05-17 21:05:15,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b8c50 type=client open_count=1 channel=[id: 0x22c231e9, 0.0.0.0/0.0.0.0:46088] timeout=300.0>) 2021-05-17 21:05:17,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b8c51 type=client open_count=1 channel=[id: 0xd8821fb6, 0.0.0.0/0.0.0.0:48600] timeout=300.0>) 2021-05-17 21:05:19,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b8c52 type=client open_count=1 channel=[id: 0x5bb031f9, 0.0.0.0/0.0.0.0:56375] timeout=300.0>) 2021-05-17 21:05:43,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b8c53 type=client open_count=1 channel=[id: 0x7dfc58b9, 0.0.0.0/0.0.0.0:50876] timeout=300.0>) 2021-05-17 21:05:45,882 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b8c54 type=client open_count=1 channel=[id: 0xd24501b8, 0.0.0.0/0.0.0.0:54360] timeout=300.0>) 2021-05-17 21:05:47,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b8c55 type=client open_count=1 channel=[id: 0xe07bea27, 0.0.0.0/0.0.0.0:33341] timeout=300.0>) 2021-05-17 21:06:12,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b8c56 type=client open_count=1 channel=[id: 0x533a6634, 0.0.0.0/0.0.0.0:38855] timeout=300.0>) 2021-05-17 21:06:14,137 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b8c57 type=client open_count=1 channel=[id: 0xb997b52e, 0.0.0.0/0.0.0.0:41617] timeout=300.0>) 2021-05-17 21:06:16,154 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b8c58 type=client open_count=1 channel=[id: 0x47ec05f3, 0.0.0.0/0.0.0.0:55355] timeout=300.0>) 2021-05-17 21:06:40,561 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b8c59 type=client open_count=1 channel=[id: 0x7c90ecaf, 0.0.0.0/0.0.0.0:50947] timeout=300.0>) 2021-05-17 21:06:42,578 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b8c5a type=client open_count=1 channel=[id: 0x728190f6, 0.0.0.0/0.0.0.0:34701] timeout=300.0>) 2021-05-17 21:06:44,598 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b8c5b type=client open_count=1 channel=[id: 0x8653b5a8, 0.0.0.0/0.0.0.0:43630] timeout=300.0>) 2021-05-17 21:07:08,739 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b8c5c type=client open_count=1 channel=[id: 0xb0e1bdc1, 0.0.0.0/0.0.0.0:42279] timeout=300.0>) 2021-05-17 21:07:10,756 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b8c5d type=client open_count=1 channel=[id: 0x2db15d2d, 0.0.0.0/0.0.0.0:46225] timeout=300.0>) 2021-05-17 21:07:12,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 21:07:29,684 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 21:07:43,394 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 21:07:43,418 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #16 test_data_load_collections_with_rebalance_out ========= 2021-05-17 21:07:43,457 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #16 test_data_load_collections_with_rebalance_out ========= 2021-05-17 21:07:56,765 | test | INFO | pool-18-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.181669694495 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 21:07:56,788 | test | INFO | pool-18-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 21:08:06,884 | test | INFO | pool-18-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 21:08:06,913 | test | INFO | pool-18-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1480000019 sec 2021-05-17 21:08:07,186 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 21:08:52,066 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 21:08:52,427 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 728803208 | 61938296 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 719073936 | 94989645 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 1088985304 | 26984417 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 21:08:52,428 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 21:09:38,203 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 21:09:38,213 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 21:25:28,105 | test | INFO | pool-18-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.............................................................................................*.......****X | 200.320 | | 21.610 | .........*****X | 21.610 | | 0.000 | *********************************X | 1.500 | | 0.000 | ***********......*****X | 13.101 | | 4.501 | .*****X | 4.501 | | 0.300 | X | 0.300 | | 0.000 | **************************X | 1.500 | | 0.000 | **********************************.****X | 3.000 | | 3.000 | X | 3.000 | | 3.000 | ****X | 3.000 | | 2.400 | X | 2.400 | | 0.000 | **********X | 1.500 | | 1.400 | ******.****X | 3.101 | | 0.900 | X | 0.900 | | 0.000 | **********X | 1.500 | | 0.000 | **********X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | ****X | 1.500 | | 1.200 | X | 1.200 | | 0.000 | *****...*****X | 6.901 | | 1.100 | X | 1.100 | | 0.600 | ***************X | 1.500 | | 0.000 | ****************************X | 1.500 | | 0.000 | **********************X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | *****.****X | 2.600 | | 2.000 | X | 2.000 | | 0.400 | ****X | 0.400 | | 0.000 | **********X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | *********************X | 1.553 | | 0.400 | X | 0.400 | | 0.000 | **********X | 1.500 | | 0.000 | ***********.******.****X | 4.500 | | 0.000 | **************************************X | 1.498 | | 0.798 | X | 0.798 | | 0.000 | *********X | 1.499 | | 0.799 | X | 0.799 | | 0.000 | ***************X | 1.499 | | 0.000 | **************************************************X | 1.499 | | 1.199 | X | 1.199 | | 0.100 | X | 0.100 | | 0.000 | *********X | 1.499 | | 0.000 | **********************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:25:28,561 | test | INFO | pool-18-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+----------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****.............................................................................................*.......****X | 66.874 | | 20.104 | .............................X | 20.104 | | 17.108 | ........................****X | 17.108 | | 0.000 | ****************************..****X | 1.500 | | 0.000 | *****.*......*****.....*.....................****X | 22.201 | | 4.300 | .....X | 4.300 | | 3.000 | ...*****.......*****X | 7.499 | | 0.000 | ******...........****X | 7.501 | | 0.500 | X | 0.500 | | 0.000 | *****....****X | 3.000 | | 0.000 | ***********..******......*****X | 6.000 | | 0.400 | X | 0.400 | | 0.000 | ****..*..****X | 3.001 | | 2.701 | ...X | 2.701 | | 1.500 | .*****...............*****X | 12.000 | | 5.000 | ......X | 5.000 | | 4.500 | .....****.*.****X | 6.000 | | 5.999 | .......X | 5.999 | | 5.999 | .......****X | 5.999 | | 1.500 | .*****X | 1.500 | | 0.100 | X | 0.100 | | 0.000 | ****************..*****X | 1.500 | | 1.500 | .*****X | 1.500 | | 1.500 | .*****..*.......****X | 7.501 | | 3.000 | ...*****X | 3.000 | | 0.000 | *****........*...****X | 7.501 | | 3.300 | ...X | 3.300 | | 3.000 | ...*****......*****X | 7.200 | | 6.300 | ........****X | 6.300 | | 4.700 | ......X | 4.700 | | 0.000 | ******....*****....*****X | 5.700 | | 4.900 | ......****X | 4.900 | | 0.000 | ***********......*.......****X | 9.000 | | 7.200 | .........X | 7.200 | | 0.000 | *****........*****X | 6.000 | | 1.500 | .****X | 1.500 | | 1.499 | .*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ****************..******..******....******...****X | 7.501 | | 4.499 | .....*****X | 4.499 | | 1.500 | .*****X | 1.500 | | 1.500 | .****X | 1.500 | | 0.000 | *****.*.****X | 1.498 | | 0.800 | X | 0.800 | | 0.000 | ***************************....*..****X | 4.495 | | 0.000 | ****************.....*...****X | 5.995 | | 0.000 | **********************************************......****X | 4.497 | | 0.300 | X | 0.300 | | 0.000 | *****..*.........****X | 7.495 | | 1.499 | .******..****X | 2.998 | | 0.200 | X | 0.200 | | 0.000 | **********************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 21:25:29,108 | test | INFO | pool-18-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.................................*.........*****..............................................*...........***X | 779.481 | | 760.581 | ................................................................................................X | 760.581 | | 675.348 | .....................................................................................****X | 675.348 | | 631.680 | ................................................................................*****X | 631.680 | | 618.730 | ..............................................................................*****...X | 645.286 | | 635.376 | ................................................................................****X | 635.376 | | 569.894 | ........................................................................X | 569.894 | | 551.201 | .....................................................................*****............*..***X | 657.719 | | 567.640 | .......................................................................*.****X | 571.078 | | 465.349 | ..........................................................X | 465.349 | | 418.567 | ....................................................****X | 418.567 | | 385.318 | ................................................X | 385.318 | | 356.997 | ............................................****........*...........*****.*.....*****...*...****X | 594.742 | | 553.233 | .....................................................................X | 553.233 | | 535.911 | ...................................................................***X | 535.911 | | 528.317 | ..................................................................X | 528.317 | | 521.450 | .................................................................****X | 521.450 | | 459.294 | .........................................................*****........****X | 515.660 | | 508.530 | ................................................................X | 508.530 | | 497.041 | ..............................................................*****........*...***X | 581.988 | | 578.655 | .........................................................................X | 578.655 | | 517.348 | .................................................................*****......*.****X | 575.702 | | 562.899 | .......................................................................*****X | 562.899 | | 418.838 | ....................................................*****..*..*****.............*****X | 550.968 | | 516.087 | .................................................................X | 516.087 | | 501.099 | ...............................................................****....*...****X | 554.314 | | 523.655 | ..................................................................******.******..*****.X | 557.544 | | 549.333 | .....................................................................*****...*.****X | 583.622 | | 564.312 | .......................................................................X | 564.312 | | 557.112 | ......................................................................****X | 557.112 | | 532.088 | ...................................................................*****X | 532.088 | | 444.642 | ........................................................*.****......******.*.****X | 517.942 | | 465.784 | ..........................................................*****X | 465.784 | | 458.205 | .........................................................X | 458.205 | | 456.660 | .........................................................****...*.....*****...*...****X | 566.212 | | 553.564 | ......................................................................*****X | 553.564 | | 491.938 | ..............................................................*****....*.****X | 537.643 | | 521.812 | .................................................................X | 521.812 | | 519.580 | .................................................................*****....*.***X | 553.589 | | 546.930 | .....................................................................*.*****...*****X | 582.177 | | 569.766 | ........................................................................X | 569.766 | | 559.945 | ......................................................................****X | 559.945 | | 430.132 | ......................................................*****X | 430.132 | | 411.219 | ...................................................*****...............*.*********X | 535.929 | | 514.062 | ................................................................X | 514.062 | | 455.306 | .........................................................*****...................*.....****X | 646.603 | | 591.008 | ..........................................................................*****X | 591.008 | | 484.001 | .............................................................X | 484.001 | | 460.201 | ..........................................................***X | 460.201 | | 445.507 | ........................................................*****X | 448.707 | | 422.516 | .....................................................*****........*...****X | 509.799 | | 492.106 | ..............................................................X | 492.106 | | 487.675 | .............................................................*****.*****.X | 502.280 | | 483.085 | ............................................................****X | 483.085 | | 475.775 | ............................................................X | 475.775 | | 459.114 | .........................................................*****.........****X | 526.311 | | 519.506 | .................................................................X | 519.506 | | 508.887 | ................................................................*****.......*****X | 564.222 | | 542.211 | ....................................................................****X | 542.211 | | 532.379 | ...................................................................*****X | 532.379 | | 489.805 | .............................................................X | 489.805 | | 475.215 | ...........................................................***X | 475.215 | | 473.353 | ...........................................................X | 473.353 | | 464.449 | ..........................................................****X | 464.449 | | 447.940 | ........................................................*...*****............****X | 561.999 | | 517.401 | .................................................................X | 517.401 | | 504.470 | ...............................................................*****....****X | 534.142 | | 525.051 | ..................................................................X | 525.051 | | 445.053 | ........................................................*****.............*****X | 552.609 | | 526.897 | ..................................................................X | 526.897 | | 519.575 | .................................................................***X | 519.575 | | 511.617 | ................................................................X | 511.617 | | 510.927 | ................................................................*****.*.****X | 525.258 | | 486.246 | .............................................................*****..*..........****X | 579.789 | | 511.168 | ................................................................X | 511.168 | | 478.549 | ............................................................*****...*.......****X | 557.765 | | 520.925 | .................................................................****X | 520.925 | | 456.656 | .........................................................X | 456.656 | | 445.117 | ........................................................*****..........*****X | 525.159 | | 492.683 | ..............................................................*****X | 492.683 | | 469.807 | ...........................................................*****...*.......****X | 551.948 | | 540.392 | ....................................................................X | 540.392 | | 531.283 | ...................................................................*****........*.............****X | 700.465 | | 559.652 | ......................................................................*****...*.....X | 619.176 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:25:29,447 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 75.7283818921 | 6437847040 | 4312772608 | 2432012288 / 7511588864 | 101825 / 206617 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 52.5000833361 | 6437847040 | 4233568256 | 2529673216 / 7511588864 | 100064 / 202649 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 67.680589353 | 6437847040 | 4244393984 | 2493804544 / 7511588864 | 100341 / 199452 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 74.8687434372 | 6437847040 | 3921924096 | 2913857536 / 9530753024 | 101647 / 199448 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 78.1520307995 | 6437847040 | 4237479936 | 2503602176 / 7511588864 | 101218 / 202004 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 21:25:42,582 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 21:27:09,773 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 165166360 | 244878255 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162442496 | 252119760 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 448111224 | 438491319 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 21:27:09,775 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #16 test_data_load_collections_with_rebalance_out ========= 2021-05-17 21:27:19,744 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_out 2021-05-17 21:27:19,746 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-17 21:27:19,813 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 21:27:19,943 | test | INFO | pool-18-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 7.99346655778 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 7.78987016883 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 13.0481158647 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 10.8320277343 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.40452977351 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 21:27:19,973 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 21:27:28,454 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 21:27:28,773 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 21:27:33,533 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.35 2021-05-17 21:27:38,638 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 3.18 2021-05-17 21:27:43,825 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 5.27 2021-05-17 21:27:48,931 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 6.92 2021-05-17 21:27:54,046 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 10.12 2021-05-17 21:27:59,148 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 11.8 2021-05-17 21:28:04,247 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 13.66 2021-05-17 21:28:07,112 | test | INFO | pool-18-thread-28 | [table_view:display:72] Keys failed in bucket1:AdGQvlwH-25-612000:LsrOW1mg2w4e5-25-760000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:07,114 | test | INFO | pool-18-thread-28 | [table_view:display:72] Keys failed in bucket1:AdGQvlwH-25-612000:LsrOW1mg2w4e5-25-760000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-27 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-129 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:09,344 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 15.1 2021-05-17 21:28:14,440 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.88 2021-05-17 21:28:19,591 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 18.47 2021-05-17 21:28:24,721 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 20.23 2021-05-17 21:28:29,904 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 21.81 2021-05-17 21:28:34,994 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 23.16 2021-05-17 21:28:40,099 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 24.34 2021-05-17 21:28:45,191 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 25.73 2021-05-17 21:28:47,141 | test | INFO | pool-18-thread-28 | [table_view:display:72] Keys failed in bucket1:AdGQvlwH-25-612000:LsrOW1mg2w4e5-25-760000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:47,142 | test | INFO | pool-18-thread-28 | [table_view:display:72] Keys failed in bucket1:AdGQvlwH-25-612000:LsrOW1mg2w4e5-25-760000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:50,328 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 27.31 2021-05-17 21:28:53,690 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:aySUnQ-26-9000:1aFPZFGN-%k-26-129000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:53,690 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:aySUnQ-26-9000:1aFPZFGN-%k-26-129000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-265 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:28:55,453 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 28.79 2021-05-17 21:29:00,569 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 30.64 2021-05-17 21:29:05,714 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 31.98 2021-05-17 21:29:10,825 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 21:29:12,545 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:aySUnQ-26-9000:1aFPZFGN-%k-26-129000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-589 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:12,546 | test | INFO | pool-18-thread-18 | [table_view:display:72] Keys failed in bucket1:aySUnQ-26-9000:1aFPZFGN-%k-26-129000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-589 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:15,920 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 21:29:21,095 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 21:29:26,436 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 34.69 2021-05-17 21:29:28,770 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:FY7YxgwgE5T69Hp-24-75000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:28,772 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:FY7YxgwgE5T69Hp-24-75000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2936 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:31,569 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 37.27 2021-05-17 21:29:35,670 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:8steEQEwrtR7qNb-25-387000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,670 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:h4ZyjOJnL0wi4x8-24-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,670 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:0mNAMsz%-25-387000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,671 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:8steEQEwrtR7qNb-25-387000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,671 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:h4ZyjOJnL0wi4x8-24-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,671 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:5ywEZF5xIg%v-24-76000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,671 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:0mNAMsz%-25-387000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,671 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:FY7YxgwgE5T69Hp-24-75000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,673 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:5ywEZF5xIg%v-24-76000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,673 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:OSl-24-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3010 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3055 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,674 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:FY7YxgwgE5T69Hp-24-75000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:35,674 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:OSl-24-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3180 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3010 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3055 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:29:36,693 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 40.26 2021-05-17 21:29:41,765 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 42.56 2021-05-17 21:29:46,894 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 44.84 2021-05-17 21:29:52,036 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 46.66 2021-05-17 21:29:57,155 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 47.88 2021-05-17 21:30:02,270 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 49.51 2021-05-17 21:30:07,394 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 50.92 2021-05-17 21:30:12,578 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 52.39 2021-05-17 21:30:15,003 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:0mNAMsz%-25-387000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,005 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:8steEQEwrtR7qNb-25-387000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,005 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:0mNAMsz%-25-387000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,006 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in bucket2:BOquagSUS-25-183000:8steEQEwrtR7qNb-25-387000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,005 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:5ywEZF5xIg%v-24-76000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,006 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:h4ZyjOJnL0wi4x8-24-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,006 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:OSl-24-926000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,006 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in bucket2:002b-23-858000:5ywEZF5xIg%v-24-76000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,007 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:OSl-24-926000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:15,006 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in bucket2:8p4QokPOqc8VbBzC-24-748000:h4ZyjOJnL0wi4x8-24-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4156 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:17,703 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 53.58 2021-05-17 21:30:22,871 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.01 2021-05-17 21:30:26,742 | test | INFO | pool-18-thread-3 | [table_view:display:72] Keys failed in bucket2:scope2:collection1 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:27,979 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 56.95 2021-05-17 21:30:28,490 | test | INFO | pool-18-thread-5 | [table_view:display:72] Keys failed in bucket2:_default:EUGSRvFVV68Mwhf6Pybd-7-230000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:30:33,063 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 58.63 2021-05-17 21:30:38,190 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 60.2 2021-05-17 21:30:43,272 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 61.59 2021-05-17 21:30:48,355 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 63.04 2021-05-17 21:30:53,424 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 64.43 2021-05-17 21:30:55,127 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:30:58,502 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.99 2021-05-17 21:31:03,592 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 21:31:08,680 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 21:31:13,917 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.86 2021-05-17 21:31:19,002 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 68.87 2021-05-17 21:31:19,842 | test | INFO | pool-18-thread-8 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:58Oq-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:19,849 | test | INFO | pool-18-thread-5 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:1_IYxNMueyk0moW-7-271000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:20,096 | test | INFO | pool-18-thread-2 | [table_view:display:72] Keys failed in default:scope1:dR22SFDJEcJ_gJ-Nz-7-266000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:20,096 | test | INFO | pool-18-thread-22 | [table_view:display:72] Keys failed in default:scope2:d1cy-7-269000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:24,141 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 71.49 2021-05-17 21:31:29,262 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 73.57 2021-05-17 21:31:29,523 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:H2BAD_eDQ35wDLPS5tpJ-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:30,660 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3279 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:30,661 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3279 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:31,621 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:BzH-7-271000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:31,622 | test | INFO | pool-18-thread-8 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:MG2SNEjsrQ-gTsVlxVuCA-7-273000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-624 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:34,351 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 75.64 2021-05-17 21:31:35,289 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in default:scope1:j9obpGHT-7-267000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-66 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:37,450 | test | INFO | pool-18-thread-7 | [table_view:display:72] Keys failed in default:m6VDuZ%hyKJSUr-27-26000:CdSfcHTCm-27-177000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3626 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:37,451 | test | INFO | pool-18-thread-7 | [table_view:display:72] Keys failed in default:m6VDuZ%hyKJSUr-27-26000:CdSfcHTCm-27-177000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3626 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:39,500 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 77.4 2021-05-17 21:31:42,861 | test | INFO | pool-18-thread-19 | [table_view:display:72] Keys failed in default:scope1:SH4T39Wak-7-266000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-634 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:44,614 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 79.1 2021-05-17 21:31:45,424 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3538 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:45,426 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3538 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:49,280 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:H2BAD_eDQ35wDLPS5tpJ-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:49,309 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:BzH-7-271000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:49,414 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in default:scope1:j9obpGHT-7-267000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:49,706 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 80.43 2021-05-17 21:31:51,065 | test | INFO | pool-18-thread-8 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:MG2SNEjsrQ-gTsVlxVuCA-7-273000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-748 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-762 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:51,555 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in default:scope1:j9obpGHT-7-267000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-328 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-307 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-342 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-297 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:53,812 | test | INFO | pool-18-thread-7 | [table_view:display:72] Keys failed in default:m6VDuZ%hyKJSUr-27-26000:CdSfcHTCm-27-177000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:53,813 | test | INFO | pool-18-thread-7 | [table_view:display:72] Keys failed in default:m6VDuZ%hyKJSUr-27-26000:CdSfcHTCm-27-177000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3944 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:54,822 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 81.87 2021-05-17 21:31:56,746 | test | INFO | pool-18-thread-19 | [table_view:display:72] Keys failed in default:scope1:SH4T39Wak-7-266000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-817 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-836 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-799 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-876 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-844 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-855 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-787 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-870 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:31:59,917 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 83.42 2021-05-17 21:32:00,970 | test | INFO | pool-18-thread-22 | [table_view:display:72] Keys failed in default:_default:MBgndZ8CtocEWsUMYc900-7-264000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-513 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:03,497 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in default:scope1:j9obpGHT-7-267000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-946 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:03,497 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:H2BAD_eDQ35wDLPS5tpJ-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-946 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:05,030 | test | INFO | pool-18-thread-3 | [table_view:display:72] Keys failed in default:scope1:De-7-265000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-513 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:05,059 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 84.91 2021-05-17 21:32:05,388 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:05,390 | test | INFO | pool-18-thread-12 | [table_view:display:72] Keys failed in default:J-26-650000:7PxAoKD8-26-802000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3665 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:10,164 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 86.38 2021-05-17 21:32:12,342 | test | INFO | pool-18-thread-9 | [table_view:display:72] Keys failed in default:scope1:j9obpGHT-7-267000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-463 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-452 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-483 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:15,026 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-625 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-500 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:15,289 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 87.79 2021-05-17 21:32:16,026 | test | INFO | pool-18-thread-2 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-157 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-179 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-81 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-123 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-195 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:17,651 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in default:_default:M-7-264000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-614 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-625 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-500 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:17,651 | test | INFO | pool-18-thread-20 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:Jz8D8hmk4LF-7-272000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-625 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-500 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:18,645 | test | INFO | pool-18-thread-15 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:BzH-7-271000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:20,242 | test | INFO | pool-18-thread-22 | [table_view:display:72] Keys failed in default:_default:MBgndZ8CtocEWsUMYc900-7-264000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-807 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-814 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-845 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-800 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-851 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:20,404 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 89.19 2021-05-17 21:32:23,216 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:Bg_-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:25,549 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 90.96 2021-05-17 21:32:25,677 | test | INFO | pool-18-thread-4 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:xtVDa8YoZBIc-v-27-660000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2752 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:25,678 | test | INFO | pool-18-thread-4 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:xtVDa8YoZBIc-v-27-660000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2752 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:28,361 | test | INFO | pool-18-thread-14 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:mSESPNgTJgmsKEs9-%WnW-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-628 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-587 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:29,732 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-899 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:30,561 | test | INFO | pool-18-thread-3 | [table_view:display:72] Keys failed in default:scope1:De-7-265000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-897 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:30,634 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 92.75 2021-05-17 21:32:32,424 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:IrtkTBmTbS_-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:32,563 | test | INFO | pool-18-thread-16 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:gy30P0jI_M0-7-276000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-94 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-126 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:35,716 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 94.29 2021-05-17 21:32:36,243 | test | INFO | pool-18-thread-20 | [table_view:display:72] Keys failed in default:tIb3mKiEAfS-7-271000:Jz8D8hmk4LF-7-272000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-767 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:36,316 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:Bg_-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-714 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-782 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:36,559 | test | INFO | pool-18-thread-2 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-206 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:38,868 | test | INFO | pool-18-thread-11 | [table_view:display:72] Keys failed in default:_default:M-7-264000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-714 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-782 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:40,825 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 95.91 2021-05-17 21:32:44,538 | test | INFO | pool-18-thread-14 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:mSESPNgTJgmsKEs9-%WnW-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-758 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-751 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:45,470 | test | INFO | pool-18-thread-4 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:xtVDa8YoZBIc-v-27-660000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2929 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:45,470 | test | INFO | pool-18-thread-4 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:xtVDa8YoZBIc-v-27-660000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2929 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:45,937 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 97.65 2021-05-17 21:32:49,095 | test | INFO | pool-18-thread-10 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:IrtkTBmTbS_-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-315 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-256 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-285 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:51,076 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 99.16 2021-05-17 21:32:51,694 | test | INFO | pool-18-thread-13 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:Bg_-7-274000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-900 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:52,513 | test | INFO | pool-18-thread-26 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-945 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-961 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:54,503 | test | INFO | pool-18-thread-3 | [table_view:display:72] Keys failed in default:scope1:De-7-265000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-902 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-900 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:55,118 | test | INFO | pool-18-thread-24 | [table_view:display:72] Keys failed in default:8KuvHd1-7-274000:XdB6fil66G-7-275000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-647 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-538 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-601 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-589 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-510 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:32:56,183 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 21:33:01,256 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 21:33:02,770 | test | INFO | pool-18-thread-2 | [table_view:display:72] Keys failed in default:_default:bL0NtInf6JTdkf1-7-263000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-480 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 21:33:06,980 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 21:33:17,457 | test | INFO | pool-18-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 21:33:17,635 | test | INFO | pool-18-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 357.694000006 sec 2021-05-17 21:34:29,073 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 21:34:29,075 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:34:29,075 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:34:29,076 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:34:29,078 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:34:29,082 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 21:34:29,084 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 21:34:29,092 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 4 docs from 0 to 5000 2021-05-17 21:34:29,094 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 21:34:50,513 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 21:34:50,520 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 21:34:50,523 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 21:34:50,525 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 21:34:50,525 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 21:34:50,526 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 21:34:50,528 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 21:34:50,530 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-17 21:34:50,532 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-17 21:34:50,533 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 8 docs from 500 to 980 2021-05-17 21:34:50,535 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 6 docs from 500 to 980 2021-05-17 21:34:50,535 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 13 docs from 0 to 500 2021-05-17 21:34:50,536 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-17 21:34:50,536 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-17 21:34:50,540 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-17 21:34:50,540 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-17 21:34:50,542 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 15 docs from 500 to 980 2021-05-17 21:34:50,542 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 17 docs from 0 to 500 2021-05-17 21:34:50,543 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-17 21:34:50,545 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-17 21:34:50,545 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-17 21:34:50,546 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 8 docs from 0 to 500 2021-05-17 21:34:50,546 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 14 docs from 0 to 500 2021-05-17 21:34:50,548 | test | ERROR | pool-18-thread-6 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-17 21:44:23,720 | test | INFO | pool-18-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....****.*.........................***X | 90.873 | | 55.864 | .................X | 55.864 | | 43.107 | .............***........................*..................***X | 168.053 | | 151.608 | .................................................X | 151.608 | | 145.217 | ...............................................**X | 145.217 | | 142.994 | ..............................................X | 142.994 | | 139.066 | .............................................****....*****...*****............***X | 197.440 | | 186.449 | .............................................................*****........*****..............................***X | 299.327 | | 149.840 | .................................................***X | 149.840 | | 116.330 | .....................................****......*..................***X | 186.340 | | 167.866 | .......................................................****X | 168.493 | | 132.457 | ...........................................***X | 132.457 | | 126.234 | .........................................*****......**X | 144.554 | | 131.350 | ..........................................X | 131.350 | | 105.534 | ..................................***X | 105.534 | | 104.741 | .................................*..***X | 107.848 | | 2.395 | ****X | 2.395 | | 2.292 | *.X | 4.591 | | 4.591 | *X | 4.591 | | 4.391 | X | 4.391 | | 0.399 | ****.****X | 3.195 | | 2.696 | ***X | 2.696 | | 2.596 | X | 2.596 | | 0.399 | ************X | 0.901 | | 0.600 | ****X | 0.600 | | 0.000 | **************X | 1.401 | | 1.400 | ***********X | 1.401 | | 0.901 | ***********X | 2.502 | | 2.203 | X | 2.203 | | 0.200 | *****X | 0.200 | | 0.000 | ***********************X | 2.601 | | 1.501 | *****X | 1.501 | | 1.201 | *****X | 1.201 | | 1.100 | ***********X | 2.502 | | 1.501 | ****X | 1.501 | | 1.401 | X | 1.401 | | 1.401 | **********X | 2.601 | | 1.703 | X | 1.703 | | 0.901 | **********X | 1.701 | | 1.301 | X | 1.301 | | 1.000 | ****X | 1.000 | | 0.500 | ************.*****X | 4.002 | | 0.700 | *****X | 0.700 | | 0.600 | *******.****X | 4.001 | | 3.502 | X | 3.502 | | 0.500 | ************.*****X | 4.101 | | 1.801 | *****X | 1.801 | | 1.001 | X | 1.001 | | 0.901 | ****X | 0.901 | | 0.701 | X | 0.701 | | 0.000 | ***********X | 1.700 | | 1.501 | *************.****X | 3.301 | | 2.201 | X | 2.201 | | 0.400 | ******.****X | 3.801 | | 2.001 | X | 2.001 | | 0.300 | **********X | 1.200 | | 0.300 | *******.****X | 3.701 | | 3.201 | X | 3.201 | | 0.200 | ***********.******X | 3.001 | | 1.300 | ************..*****X | 6.402 | | 0.300 | *****************************X | 1.701 | | 1.501 | X | 1.501 | | 1.300 | *****************X | 1.600 | | 1.300 | ************.*****X | 3.301 | | 1.200 | *****X | 1.200 | | 0.900 | ***********X | 1.200 | | 0.800 | X | 0.800 | | 0.600 | *****.******X | 5.001 | | 1.100 | ******************X | 2.600 | | 2.500 | *****X | 2.801 | | 2.701 | X | 2.701 | | 2.601 | **X | 2.601 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:44:24,677 | test | INFO | pool-18-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *******.****.........****X | 17.607 | | 8.209 | ...X | 8.209 | | 3.198 | ***.......*...****....****..*.***X | 30.102 | | 28.685 | ................X | 28.685 | | 28.185 | ................****.....***X | 37.312 | | 32.186 | ..................*****...*****.............................***X | 84.968 | | 49.921 | .............................****...*.***X | 56.446 | | 51.341 | ..............................*.****.***X | 54.697 | | 53.845 | ...............................X | 53.845 | | 51.168 | ..............................***X | 51.168 | | 36.326 | .....................****...*.....****......****..............*****.............*.****....................................***X | 164.856 | | 64.443 | ......................................*****...........*****..................***X | 113.004 | | 103.799 | .............................................................X | 103.799 | | 80.385 | ...............................................****.......****.....***X | 99.697 | | 97.472 | ..........................................................X | 97.472 | | 89.712 | .....................................................****....*****........***X | 109.663 | | 91.831 | ......................................................******.......................................****X | 156.397 | | 134.981 | ................................................................................X | 134.981 | | 119.612 | .......................................................................*****.......*...........****X | 149.708 | | 118.123 | ......................................................................*****X | 118.123 | | 111.986 | ..................................................................******X | 111.986 | | 67.123 | .......................................*****X | 67.123 | | 62.618 | ....................................******..........*****X | 78.720 | | 60.526 | ...................................******......*****X | 70.722 | | 32.910 | ..................*****X | 32.910 | | 0.800 | *****X | 0.800 | | 0.700 | ******.*****X | 2.300 | | 1.202 | X | 1.202 | | 0.200 | ***********.*****X | 2.002 | | 1.602 | X | 1.602 | | 1.100 | ****X | 1.100 | | 0.501 | X | 0.501 | | 0.401 | ****X | 0.401 | | 0.400 | ******.*****X | 2.101 | | 0.600 | ***********X | 1.400 | | 0.801 | X | 0.801 | | 0.701 | *****X | 0.701 | | 0.500 | *****X | 0.500 | | 0.300 | ******.*****X | 3.201 | | 1.502 | X | 1.502 | | 0.000 | **********X | 1.501 | | 1.201 | X | 1.201 | | 1.001 | *****X | 1.001 | | 0.000 | ************.*****X | 3.201 | | 1.701 | X | 1.701 | | 0.600 | ******..****X | 3.301 | | 2.201 | X | 2.201 | | 0.500 | ******.****X | 1.800 | | 0.900 | X | 0.900 | | 0.400 | ****************X | 1.400 | | 1.101 | X | 1.101 | | 1.000 | ******..*****X | 3.401 | | 0.200 | ******X | 0.400 | | 0.300 | ****X | 0.300 | | 0.000 | *************..****X | 3.501 | | 2.601 | X | 2.601 | | 0.900 | *****X | 0.900 | | 0.600 | ***********X | 0.700 | | 0.500 | ******.*****X | 2.101 | | 1.600 | *****X | 1.600 | | 0.700 | ******X | 0.700 | | 0.500 | ******.*****X | 2.000 | | 1.200 | ************.*****X | 1.900 | | 0.500 | *************.***********X | 2.300 | | 1.500 | ******.*****X | 2.201 | | 1.400 | X | 1.400 | | 1.300 | ***********X | 1.600 | | 0.600 | ****X | 0.600 | +---------+--------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:44:25,703 | test | INFO | pool-18-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...................................................*............**X | 547.286 | | 428.335 | .................................................X | 428.335 | | 234.721 | ..........................****.........................................................................****X | 855.766 | | 295.416 | .................................***X | 295.416 | | 280.213 | ...............................X | 280.213 | | 268.630 | ..............................**X | 268.630 | | 268.630 | ..............................*.....*****............***X | 416.157 | | 243.304 | ...........................****X | 243.304 | | 187.239 | ....................****.............*****....................**X | 468.580 | | 468.580 | .....................................................*X | 468.580 | | 440.297 | ..................................................*X | 440.297 | | 440.297 | ..................................................*X | 440.297 | | 238.347 | ..........................***X | 238.347 | | 171.567 | ...................****.*.***X | 194.350 | | 176.155 | ...................*****.........................***X | 392.245 | | 259.592 | .............................*****..****..*....*X | 325.451 | | 325.451 | .....................................*X | 325.451 | | 188.000 | ....................*****.........................*..............****...................................***X | 817.075 | | 523.122 | ............................................................X | 523.122 | | 523.122 | ............................................................**X | 523.122 | | 432.604 | .................................................*..***X | 449.310 | | 227.489 | .........................*****.................................***X | 507.723 | | 242.123 | ...........................***X | 242.123 | | 194.609 | .....................***X | 194.609 | | 192.694 | .....................X | 192.694 | | 192.206 | .....................***X | 192.206 | | 160.641 | .................***X | 160.641 | | 144.051 | ...............X | 144.051 | | 109.557 | ...........****..........******............*X | 299.353 | | 299.353 | .................................**X | 299.353 | | 292.526 | .................................******..............*****X | 413.557 | | 270.211 | ..............................*****X | 270.211 | | 241.937 | ...........................******......*****X | 294.295 | | 288.575 | ................................******.......*....***X | 377.520 | | 377.520 | ...........................................X | 377.520 | | 301.021 | ..................................*****X | 301.021 | | 288.572 | ................................****X | 288.572 | | 288.572 | ................................*............................X | 527.685 | | 527.685 | ............................................................****X | 527.685 | | 445.741 | ...................................................X | 445.741 | | 417.108 | ...............................................*****..........*...*X | 524.595 | | 524.595 | ............................................................***X | 524.912 | | 506.670 | ..........................................................*****.......*....****X | 603.747 | | 451.118 | ...................................................X | 451.118 | | 356.307 | ........................................****X | 356.307 | | 355.894 | ........................................*......****X | 403.533 | | 396.893 | .............................................X | 396.893 | | 371.316 | ..........................................*****...............*..........****X | 583.817 | | 433.718 | .................................................X | 433.718 | | 342.033 | ......................................*****...*.......****X | 421.504 | | 412.158 | ...............................................*.******...*****X | 452.604 | | 402.002 | .............................................*****X | 402.002 | | 338.338 | ......................................******.................*****X | 486.260 | | 449.413 | ...................................................X | 449.413 | | 413.342 | ...............................................****X | 413.342 | | 358.599 | ........................................X | 358.599 | | 337.968 | ......................................*****..*..*.*****.....*****X | 420.041 | | 403.201 | ..............................................X | 403.201 | | 383.929 | ...........................................*****...X | 405.995 | | 391.975 | ............................................****X | 391.975 | | 327.471 | .....................................******.........*..X | 424.136 | | 424.136 | ................................................***X | 424.136 | | 386.518 | ............................................X | 386.518 | | 355.244 | ........................................X | 355.244 | | 332.025 | .....................................****X | 332.025 | | 311.934 | ...................................******...*.****X | 347.836 | | 346.241 | .......................................X | 346.241 | | 336.433 | ......................................*****...*.****X | 370.018 | | 357.160 | ........................................*****X | 357.160 | | 339.799 | ......................................******..*****X | 352.743 | | 339.582 | ......................................X | 339.582 | | 313.776 | ...................................*****.....******.******.......*****X | 423.773 | | 290.433 | ................................******.......******.*.....****X | 394.419 | | 382.127 | ...........................................*.****X | 386.178 | | 325.508 | .....................................******....*.****X | 375.718 | | 360.172 | .........................................******....******.....******.X | 447.496 | | 403.658 | ..............................................****X | 403.658 | | 392.700 | ............................................*..****X | 402.968 | | 396.193 | .............................................******........X | 464.623 | | 453.466 | ...................................................*****X | 453.466 | | 432.655 | .................................................*..*****.......*****X | 509.547 | | 449.004 | ...................................................X | 449.004 | | 400.749 | .............................................****X | 400.749 | | 393.335 | ............................................X | 393.335 | | 381.561 | ...........................................*****......*....******.....*****X | 513.383 | | 485.281 | .......................................................X | 485.281 | | 461.048 | ....................................................*****.........******..*.....****X | 597.484 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 21:44:25,749 | test | WARNING | Thread-39884 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BOquagSUS-25-183000:8steEQEwrtR7qNb-25-387000 create failed keys from task: [u'test_collections-3180', u'test_collections-3010', u'test_collections-4156', u'test_collections-3055'] 2021-05-17 21:44:25,750 | test | WARNING | Thread-39885 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:BOquagSUS-25-183000:0mNAMsz%-25-387000 create failed keys from task: [u'test_collections-3180', u'test_collections-3010', u'test_collections-4156', u'test_collections-3055'] 2021-05-17 21:44:25,756 | test | WARNING | Thread-39915 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:EUGSRvFVV68Mwhf6Pybd-7-230000 delete failed keys from task: [u'test_collections-1'] 2021-05-17 21:44:25,766 | test | WARNING | Thread-39956 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:8p4QokPOqc8VbBzC-24-748000:OSl-24-926000 create failed keys from task: [u'test_collections-3180', u'test_collections-3010', u'test_collections-4156', u'test_collections-3055'] 2021-05-17 21:44:25,766 | test | WARNING | Thread-39957 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:8p4QokPOqc8VbBzC-24-748000:h4ZyjOJnL0wi4x8-24-925000 create failed keys from task: [u'test_collections-3180', u'test_collections-3010', u'test_collections-4156', u'test_collections-3055'] 2021-05-17 21:44:25,805 | test | WARNING | Thread-40109 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:aySUnQ-26-9000:1aFPZFGN-%k-26-129000 create failed keys from task: [u'test_collections-419', u'test_collections-589', u'test_collections-265'] 2021-05-17 21:44:25,806 | test | WARNING | Thread-40112 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:AdGQvlwH-25-612000:LsrOW1mg2w4e5-25-760000 create failed keys from task: [u'test_collections-27', u'test_collections-129', u'test_collections-654'] 2021-05-17 21:44:25,835 | test | WARNING | Thread-40195 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:bL0NtInf6JTdkf1-7-263000 update failed keys from task: [u'test_collections-625', u'test_collections-945', u'test_collections-921', u'test_collections-500', u'test_collections-961', u'test_collections-614', u'test_collections-658', u'test_collections-856', u'test_collections-934', u'test_collections-810', u'test_collections-899', u'test_collections-599', u'test_collections-971', u'test_collections-651', u'test_collections-770'] 2021-05-17 21:44:25,842 | test | WARNING | Thread-40192 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:MBgndZ8CtocEWsUMYc900-7-264000 update failed keys from task: [u'test_collections-807', u'test_collections-669', u'test_collections-845', u'test_collections-846', u'test_collections-800', u'test_collections-880', u'test_collections-814', u'test_collections-837', u'test_collections-513', u'test_collections-579', u'test_collections-556', u'test_collections-872', u'test_collections-851', u'test_collections-770'] 2021-05-17 21:44:25,848 | test | WARNING | Thread-40230 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:H2BAD_eDQ35wDLPS5tpJ-7-274000 update failed keys from task: [u'test_collections-847', u'test_collections-803', u'test_collections-946', u'test_collections-648', u'test_collections-967', u'test_collections-741', u'test_collections-593'] 2021-05-17 21:44:25,849 | test | WARNING | Thread-40233 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:Bg_-7-274000 update failed keys from task: [u'test_collections-900', u'test_collections-714', u'test_collections-810', u'test_collections-921', u'test_collections-567', u'test_collections-773', u'test_collections-782'] 2021-05-17 21:44:25,878 | test | WARNING | Thread-40198 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:M-7-264000 update failed keys from task: [u'test_collections-849', u'test_collections-614', u'test_collections-625', u'test_collections-658', u'test_collections-714', u'test_collections-758', u'test_collections-679', u'test_collections-500', u'test_collections-782', u'test_collections-781'] 2021-05-17 21:44:25,890 | test | WARNING | Thread-39959 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:002b-23-858000:5ywEZF5xIg%v-24-76000 create failed keys from task: [u'test_collections-3180', u'test_collections-3010', u'test_collections-4156', u'test_collections-3055'] 2021-05-17 21:44:25,894 | test | WARNING | Thread-39958 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:002b-23-858000:FY7YxgwgE5T69Hp-24-75000 create failed keys from task: [u'test_collections-3180', u'test_collections-2936', u'test_collections-3010', u'test_collections-3055'] 2021-05-17 21:44:25,934 | test | WARNING | Thread-40362 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:d1cy-7-269000 update failed keys from task: [u'test_collections-927', u'test_collections-962'] 2021-05-17 21:44:25,938 | test | WARNING | Thread-40241 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:IrtkTBmTbS_-7-274000 delete failed keys from task: [u'test_collections-126', u'test_collections-203', u'test_collections-125', u'test_collections-243', u'test_collections-285', u'test_collections-1', u'test_collections-2', u'test_collections-44', u'test_collections-88', u'test_collections-315', u'test_collections-256', u'test_collections-278', u'test_collections-133', u'test_collections-393'] 2021-05-17 21:44:25,960 | test | WARNING | Thread-40008 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope2:collection1 delete failed keys from task: [u'test_collections-1'] 2021-05-17 21:44:26,002 | test | WARNING | Thread-40336 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:SH4T39Wak-7-266000 update failed keys from task: [u'test_collections-817', u'test_collections-836', u'test_collections-634', u'test_collections-799', u'test_collections-876', u'test_collections-844', u'test_collections-855', u'test_collections-787', u'test_collections-870'] 2021-05-17 21:44:26,015 | test | WARNING | Thread-40248 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:mSESPNgTJgmsKEs9-%WnW-7-274000 update failed keys from task: [u'test_collections-708', u'test_collections-718', u'test_collections-628', u'test_collections-758', u'test_collections-657', u'test_collections-797', u'test_collections-567', u'test_collections-751', u'test_collections-587', u'test_collections-781'] 2021-05-17 21:44:26,033 | test | WARNING | Thread-40249 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:gy30P0jI_M0-7-276000 delete failed keys from task: [u'test_collections-1', u'test_collections-2', u'test_collections-44', u'test_collections-88', u'test_collections-94', u'test_collections-126', u'test_collections-125', u'test_collections-133'] 2021-05-17 21:44:26,033 | test | WARNING | Thread-40250 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:xtVDa8YoZBIc-v-27-660000 create failed keys from task: [u'test_collections-2929', u'test_collections-2752'] 2021-05-17 21:44:26,036 | test | WARNING | Thread-40261 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:XdB6fil66G-7-275000 update failed keys from task: [u'test_collections-647', u'test_collections-538', u'test_collections-601', u'test_collections-589', u'test_collections-510'] 2021-05-17 21:44:26,164 | test | WARNING | Thread-40266 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:J-26-650000:7PxAoKD8-26-802000 create failed keys from task: [u'test_collections-3538', u'test_collections-3665', u'test_collections-3279'] 2021-05-17 21:44:26,187 | test | WARNING | Thread-40269 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:tIb3mKiEAfS-7-271000:Jz8D8hmk4LF-7-272000 update failed keys from task: [u'test_collections-838', u'test_collections-614', u'test_collections-625', u'test_collections-658', u'test_collections-735', u'test_collections-856', u'test_collections-810', u'test_collections-679', u'test_collections-767', u'test_collections-500'] 2021-05-17 21:44:26,219 | test | WARNING | Thread-40309 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:dR22SFDJEcJ_gJ-Nz-7-266000 update failed keys from task: [u'test_collections-927', u'test_collections-962'] 2021-05-17 21:44:26,236 | test | WARNING | Thread-40315 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:j9obpGHT-7-267000 update failed keys from task: [u'test_collections-729', u'test_collections-803', u'test_collections-946', u'test_collections-967', u'test_collections-741', u'test_collections-760'] 2021-05-17 21:44:26,298 | test | WARNING | Thread-40281 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:tIb3mKiEAfS-7-271000:BzH-7-271000 update failed keys from task: [u'test_collections-729', u'test_collections-803', u'test_collections-958', u'test_collections-955', u'test_collections-922', u'test_collections-648', u'test_collections-910', u'test_collections-785', u'test_collections-741'] 2021-05-17 21:44:26,302 | test | WARNING | Thread-40318 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:De-7-265000 update failed keys from task: [u'test_collections-902', u'test_collections-669', u'test_collections-856', u'test_collections-900', u'test_collections-513', u'test_collections-579', u'test_collections-810', u'test_collections-556', u'test_collections-897'] 2021-05-17 21:44:26,381 | test | WARNING | Thread-40259 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8KuvHd1-7-274000:58Oq-7-274000 update failed keys from task: [u'test_collections-962'] 2021-05-17 21:44:26,405 | test | WARNING | Thread-40294 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:tIb3mKiEAfS-7-271000:MG2SNEjsrQ-gTsVlxVuCA-7-273000 update failed keys from task: [u'test_collections-729', u'test_collections-748', u'test_collections-648', u'test_collections-876', u'test_collections-624', u'test_collections-820', u'test_collections-762', u'test_collections-760'] 2021-05-17 21:44:26,569 | test | WARNING | Thread-40267 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:m6VDuZ%hyKJSUr-27-26000:CdSfcHTCm-27-177000 create failed keys from task: [u'test_collections-3944', u'test_collections-3626'] 2021-05-17 21:44:26,818 | test | WARNING | Thread-40315 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:j9obpGHT-7-267000 delete failed keys from task: [u'test_collections-328', u'test_collections-307', u'test_collections-348', u'test_collections-342', u'test_collections-463', u'test_collections-266', u'test_collections-483', u'test_collections-361', u'test_collections-66', u'test_collections-438', u'test_collections-455', u'test_collections-452', u'test_collections-297'] 2021-05-17 21:44:26,838 | test | WARNING | Thread-40289 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:tIb3mKiEAfS-7-271000:1_IYxNMueyk0moW-7-271000 update failed keys from task: [u'test_collections-962'] 2021-05-17 21:44:27,028 | test | WARNING | Thread-40195 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:bL0NtInf6JTdkf1-7-263000 delete failed keys from task: [u'test_collections-98', u'test_collections-206', u'test_collections-468', u'test_collections-81', u'test_collections-166', u'test_collections-123', u'test_collections-385', u'test_collections-480', u'test_collections-419', u'test_collections-215', u'test_collections-157', u'test_collections-179', u'test_collections-334', u'test_collections-274', u'test_collections-396', u'test_collections-195', u'test_collections-7'] 2021-05-17 21:44:35,855 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 21:45:26,737 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 21:45:26,739 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 21:45:50,017 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 21:45:50,019 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 21:46:04,686 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 21:46:04,687 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 21:47:13,121 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.128000021 seconds to complete 2021-05-17 21:47:13,137 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2548.803s OK During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 16 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_16 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_17 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_17', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 17, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 21:47:22,506 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #17 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 21:47:23,233 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #17 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 21:47:23,657 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x515d13 type=client open_count=1 channel=[id: 0x76768e97, 0.0.0.0/0.0.0.0:32808] timeout=300.0>) 2021-05-17 21:47:44,460 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x515d14 type=client open_count=1 channel=[id: 0x58674f17, 0.0.0.0/0.0.0.0:41969] timeout=300.0>) 2021-05-17 21:47:46,477 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x515d15 type=client open_count=1 channel=[id: 0xbaed3584, 0.0.0.0/0.0.0.0:33062] timeout=300.0>) 2021-05-17 21:47:48,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x515d16 type=client open_count=1 channel=[id: 0x656ddbb5, 0.0.0.0/0.0.0.0:40819] timeout=300.0>) 2021-05-17 21:48:12,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x515d17 type=client open_count=1 channel=[id: 0xd01ddd0d, 0.0.0.0/0.0.0.0:37535] timeout=300.0>) 2021-05-17 21:48:14,599 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x515d18 type=client open_count=1 channel=[id: 0x44f6f883, 0.0.0.0/0.0.0.0:47809] timeout=300.0>) 2021-05-17 21:48:16,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x515d19 type=client open_count=1 channel=[id: 0xd9a64526, 0.0.0.0/0.0.0.0:32971] timeout=300.0>) 2021-05-17 21:48:40,522 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x515d1a type=client open_count=1 channel=[id: 0xed4fc691, 0.0.0.0/0.0.0.0:50860] timeout=300.0>) 2021-05-17 21:48:42,543 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x515d1b type=client open_count=1 channel=[id: 0x3b845590, 0.0.0.0/0.0.0.0:37885] timeout=300.0>) 2021-05-17 21:48:44,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x515d1c type=client open_count=1 channel=[id: 0x7eb61590, 0.0.0.0/0.0.0.0:56995] timeout=300.0>) 2021-05-17 21:49:08,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x515d1d type=client open_count=1 channel=[id: 0xf16c1b50, 0.0.0.0/0.0.0.0:58658] timeout=300.0>) 2021-05-17 21:49:10,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x515d1e type=client open_count=1 channel=[id: 0x509f7ca1, 0.0.0.0/0.0.0.0:37193] timeout=300.0>) 2021-05-17 21:49:12,536 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x515d1f type=client open_count=1 channel=[id: 0xc51f8ad6, 0.0.0.0/0.0.0.0:55485] timeout=300.0>) 2021-05-17 21:49:36,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x515d20 type=client open_count=1 channel=[id: 0x52293037, 0.0.0.0/0.0.0.0:40616] timeout=300.0>) 2021-05-17 21:49:38,631 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x515d21 type=client open_count=1 channel=[id: 0x684284e9, 0.0.0.0/0.0.0.0:45422] timeout=300.0>) 2021-05-17 21:49:40,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 21:49:57,967 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 21:50:12,394 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 21:50:13,434 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #17 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 21:50:13,500 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #17 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 21:50:24,973 | test | INFO | pool-19-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 9.2175419888 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 21:50:25,000 | test | INFO | pool-19-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 21:50:35,111 | test | INFO | pool-19-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 21:50:35,155 | test | INFO | pool-19-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1819999218 sec 2021-05-17 21:50:35,526 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 21:51:27,753 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 21:51:28,104 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 657500672 | 91510909 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 647489296 | 109557273 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 740925872 | 66413932 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 21:51:28,105 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 21:52:05,802 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 21:52:05,812 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 22:09:50,161 | test | INFO | pool-19-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****...........................................*............................................*****............*****X | 826.034 | | 716.940 | .....................................................................................X | 716.940 | | 648.554 | .............................................................................****X | 648.554 | | 632.269 | ...........................................................................X | 632.269 | | 560.428 | ..................................................................*****.........******.........******..*..****X | 739.796 | | 553.737 | ..................................................................*****X | 553.737 | | 448.637 | .....................................................******X | 448.785 | | 431.775 | ...................................................****X | 431.775 | | 420.773 | .................................................******....*****....*....*****X | 516.208 | | 501.593 | ...........................................................*****X | 501.593 | | 431.780 | ...................................................*****.....*...........****X | 563.715 | | 527.438 | ..............................................................X | 527.438 | | 482.605 | .........................................................******......******.....****X | 573.956 | | 570.909 | ....................................................................*****X | 570.909 | | 509.281 | ............................................................X | 509.281 | | 468.081 | .......................................................******.*****X | 477.280 | | 440.552 | ....................................................*****X | 442.752 | | 411.385 | ................................................****X | 411.385 | | 408.785 | ................................................*.........*****...*****X | 505.788 | | 451.852 | .....................................................************.....*..****X | 504.327 | | 502.231 | ...........................................................*****X | 502.231 | | 476.211 | ........................................................****X | 476.211 | | 419.034 | .................................................******.******..............*****X | 542.862 | | 509.239 | ............................................................X | 509.239 | | 501.370 | ...........................................................****X | 501.370 | | 482.218 | .........................................................*****X | 482.218 | | 430.376 | ...................................................***********X | 435.328 | | 404.440 | ...............................................*****X | 404.440 | | 390.846 | ..............................................******..........*****X | 478.405 | | 416.900 | .................................................*****.....*....****X | 495.238 | | 465.277 | .......................................................******.******X | 472.765 | | 446.179 | .....................................................*****....*.****X | 489.101 | | 411.699 | ................................................*****.*....************......*****X | 501.636 | | 480.269 | .........................................................****X | 480.269 | | 419.365 | .................................................X | 419.365 | | 402.800 | ...............................................*****....X | 434.301 | | 414.974 | .................................................*****...........*****X | 509.419 | | 488.528 | ..........................................................*****X | 488.528 | | 406.893 | ................................................******..X | 422.097 | | 398.384 | ...............................................******.****X | 405.546 | | 385.180 | .............................................*..*****........*...*****....X | 520.848 | | 516.957 | .............................................................****X | 516.957 | | 409.472 | ................................................******..........******..*****X | 505.274 | | 485.311 | .........................................................*****X | 485.311 | | 477.259 | ........................................................*****X | 477.259 | | 392.688 | ..............................................X | 392.688 | | 367.887 | ...........................................*****........*..****X | 453.116 | | 443.516 | ....................................................X | 443.516 | | 427.907 | ..................................................*****......*****X | 476.714 | | 467.721 | .......................................................*****X | 467.721 | | 362.208 | ..........................................******.******....*****X | 400.434 | | 384.533 | .............................................*****..*.......****X | 455.506 | | 441.820 | ....................................................X | 441.820 | | 405.531 | ................................................****X | 405.531 | | 397.323 | ...............................................X | 397.323 | | 397.174 | ...............................................*****.............*****X | 503.923 | | 390.578 | ..............................................******....*****.*.......****X | 488.260 | | 485.254 | .........................................................X | 485.254 | | 475.135 | ........................................................****X | 475.135 | | 470.654 | .......................................................X | 470.654 | | 467.668 | .......................................................*****....*****X | 497.678 | | 458.323 | ......................................................*****X | 458.323 | | 448.290 | .....................................................X | 448.290 | | 445.891 | ....................................................****X | 445.891 | | 429.620 | ...................................................*****X | 432.525 | | 390.093 | ..............................................*****X | 390.093 | | 377.895 | ............................................******....******......****X | 461.999 | | 450.488 | .....................................................X | 450.488 | | 435.079 | ...................................................X | 435.079 | | 413.143 | .................................................****....*...*****....X | 508.622 | | 501.570 | ...........................................................****X | 501.570 | | 468.395 | .......................................................*..****X | 479.990 | | 468.212 | .......................................................X | 468.212 | | 444.350 | ....................................................*****.X | 452.817 | | 415.871 | .................................................****X | 415.871 | | 396.640 | ...............................................X | 396.640 | | 363.500 | ...........................................*****...*......***X | 442.446 | | 435.325 | ...................................................*......***********..X | 496.198 | | 486.579 | .........................................................*****...*****X | 511.142 | | 309.703 | ....................................*X | 309.703 | +---------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 22:09:51,269 | test | INFO | pool-19-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.........................................................................*...........................****X | 26.974 | | 7.175 | .........................X | 7.175 | | 0.000 | ************************************..*...*****.......*......*****...........*.....................*****.X | 13.788 | | 11.589 | .........................................****X | 11.589 | | 9.893 | ...................................X | 9.893 | | 8.994 | ................................****X | 8.994 | | 0.000 | ******.....*......****X | 2.998 | | 1.598 | ....X | 1.598 | | 0.000 | *****************************************...........******..........*............*****.................................*****X | 17.993 | | 11.995 | ...........................................*****X | 11.995 | | 0.000 | *****................*****X | 4.497 | | 0.000 | ******************************...........................****X | 7.497 | | 2.999 | ..........*****X | 2.999 | | 2.997 | ..........*****X | 2.997 | | 0.000 | ******...........*****X | 2.999 | | 0.000 | ******................................******............*****X | 12.098 | | 0.000 | ******..............******....*****X | 4.998 | | 0.000 | ******...........******...........................*****X | 10.499 | | 0.000 | ******................******........*...****X | 7.499 | | 4.200 | ..............X | 4.200 | | 3.000 | ..........****X | 3.000 | | 0.000 | ******...........*****X | 3.000 | | 0.000 | ************...........*****X | 3.079 | | 0.000 | ************.....******............*......****X | 6.399 | | 1.100 | ...*****X | 1.100 | | 0.900 | ..*.****X | 1.302 | | 0.602 | .X | 0.602 | | 0.200 | *****.....******...........****X | 4.504 | | 3.304 | ...........X | 3.304 | | 0.000 | *****************..............................................................................................*****X | 25.500 | | 9.022 | ................................X | 9.022 | | 3.000 | ..........****X | 3.000 | | 2.200 | .......X | 2.200 | | 0.000 | *****.*....****X | 1.500 | | 0.700 | .X | 0.700 | | 0.000 | ***********..*...*****...*...*****.....*****X | 4.500 | | 2.900 | .........X | 2.900 | | 1.500 | ....****X | 1.500 | | 0.000 | *****.*....****X | 1.500 | | 0.700 | .X | 0.700 | | 0.000 | *****............*....****X | 4.500 | | 3.400 | ...........X | 3.400 | | 2.999 | ..........****X | 2.999 | | 0.000 | ******.....*****X | 1.500 | | 0.000 | ******.....*****...*.....*.....***X | 4.901 | | 3.801 | .............X | 3.801 | | 2.599 | ........****X | 2.599 | | 0.401 | X | 0.401 | | 0.000 | *****.....************......*****X | 3.000 | | 0.000 | *****..*...****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | *****************....*.****......*..........*......***X | 7.500 | | 7.499 | ..........................*X | 7.499 | | 7.499 | ..........................***X | 7.499 | | 4.300 | ..............X | 4.300 | | 1.500 | ....****X | 1.500 | | 0.000 | *************************X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 22:09:52,315 | test | INFO | pool-19-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.......................*............................****X | 8.991 | | 4.792 | ..........................X | 4.792 | | 0.000 | *****************************.*.......******************........................*****...X | 6.295 | | 6.095 | .................................****X | 6.095 | | 4.697 | .........................*****X | 4.697 | | 0.000 | ******.................***********X | 2.998 | | 0.000 | ******************************............................................*****X | 7.896 | | 1.099 | .....******......*.......*****....................................*....................................****X | 16.094 | | 2.999 | ................*****X | 2.999 | | 0.000 | ******............................................................................*****X | 13.492 | | 0.000 | ***********************........******...................*****X | 4.898 | | 1.100 | .....******..*****X | 1.499 | | 0.400 | .X | 0.400 | | 0.000 | *****..................*.......****X | 4.499 | | 3.399 | ..................X | 3.399 | | 2.999 | ................*****X | 3.000 | | 2.999 | ................*****..................*.......****X | 7.499 | | 2.000 | ..........X | 2.000 | | 0.000 | *****........................*..........****X | 5.998 | | 2.699 | ..............X | 2.699 | | 1.499 | .......****X | 1.499 | | 0.400 | .X | 0.400 | | 0.000 | ****.................************X | 3.000 | | 0.000 | *****............................................................................*****X | 13.496 | | 8.999 | ..................................................*****X | 8.999 | | 3.000 | ................*****X | 3.000 | | 1.500 | .......***********X | 1.500 | | 0.000 | ******.............*......................................****X | 9.000 | | 0.000 | ******************......*..*****.........................*.........****X | 7.499 | | 2.000 | ..........X | 2.000 | | 0.000 | ***********......*..****X | 1.502 | | 0.000 | *****.......................*............................****X | 9.007 | | 4.809 | ..........................X | 4.809 | | 0.000 | ******************..........................................******.........*****X | 9.001 | | 0.000 | ******.........*........****X | 3.000 | | 1.799 | .........X | 1.799 | | 1.500 | .......****X | 1.500 | | 1.500 | .......****X | 1.500 | | 1.500 | .......X | 1.500 | | 1.500 | .......****X | 1.500 | | 0.700 | ..X | 0.700 | | 0.000 | *****....................................................................*****X | 12.000 | | 10.400 | ..........................................................*****X | 10.400 | | 5.001 | ...........................X | 5.001 | | 0.100 | ****X | 0.100 | | 0.000 | ******************.........................****X | 4.499 | | 2.302 | ............X | 2.302 | | 1.500 | .......****X | 1.500 | | 0.401 | .X | 0.401 | | 0.000 | *****..........*....****X | 2.599 | | 0.400 | .*****..*............*.......****.........*............****X | 7.900 | | 3.103 | ................X | 3.103 | | 1.500 | .......*****X | 1.500 | | 1.500 | .......*****.....*............****X | 4.500 | | 4.500 | ........................*****X | 4.500 | | 3.300 | .................X | 3.300 | | 1.202 | .....X | 1.202 | | 0.000 | ****.......*..........****X | 3.000 | | 0.801 | ...X | 0.801 | | 0.000 | ***********..................*.......*****.............*....***X | 7.500 | | 6.300 | ..................................X | 6.300 | | 4.201 | ......................X | 4.201 | | 3.000 | ................****.....................................*.......................*.......................***X | 17.598 | | 11.999 | ...................................................................X | 11.999 | | 6.400 | ...................................****X | 6.400 | | 0.000 | **************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 22:09:52,638 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 76.9520507992 | 6437847040 | 4211036160 | 2544754688 / 7511588864 | 126142 / 253507 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 73.439557326 | 6437847040 | 4158504960 | 2560577536 / 7511588864 | 125730 / 255052 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 72.6062101035 | 6437847040 | 3745869824 | 3091140608 / 9530753024 | 126549 / 246911 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 57.4190430159 | 6437847040 | 4181446656 | 2545283072 / 7511588864 | 126536 / 254420 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 22:10:03,252 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 22:11:16,877 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 160644352 | 265123204 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 158574080 | 283947205 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 438830000 | 449724287 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 22:11:16,878 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #17 test_data_load_collections_with_swap_rebalance ========= 2021-05-17 22:11:25,134 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during swap_rebalance 2021-05-17 22:11:25,134 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-17 22:11:28,576 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 22:11:28,756 | test | INFO | pool-19-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 14.7916666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.4528606903 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 16.4022134071 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.94816753054 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 22:11:28,937 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:11:34,145 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:11:38,298 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 22:11:39,289 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:11:44,368 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.42 2021-05-17 22:11:49,470 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 1.59 2021-05-17 22:11:54,556 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 3.1 2021-05-17 22:11:55,954 | test | INFO | pool-19-thread-6 | [table_view:display:72] Keys failed in bucket1:scope1:CxnncA4-zfnO-35-625000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:11:55,954 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in bucket1:scope1:bTWZ-kK%kNkNzU_f-35-622000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:11:59,687 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 4.71 2021-05-17 22:12:04,826 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 7.04 2021-05-17 22:12:09,959 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 9.32 2021-05-17 22:12:15,051 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 11.63 2021-05-17 22:12:20,171 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 13.82 2021-05-17 22:12:25,265 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 16.16 2021-05-17 22:12:30,392 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 18.08 2021-05-17 22:12:35,542 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 20.25 2021-05-17 22:12:40,654 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 22.35 2021-05-17 22:12:45,733 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 23.95 2021-05-17 22:12:50,855 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 25.15 2021-05-17 22:12:55,960 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 26.44 2021-05-17 22:13:01,059 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 27.99 2021-05-17 22:13:06,201 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 29.81 2021-05-17 22:13:11,341 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-17 22:13:16,444 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:13:21,546 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:13:26,664 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:13:31,877 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 35.31 2021-05-17 22:13:32,283 | test | INFO | pool-19-thread-3 | [table_view:display:72] Keys failed in bucket2:scope1:i4MQdW_-35-609000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:36,983 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 37.2 2021-05-17 22:13:42,130 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 39.53 2021-05-17 22:13:44,578 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,579 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,579 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3323 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,578 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,579 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,581 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3323 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,579 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,581 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3323 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,582 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3323 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:44,582 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3391 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3201 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3323 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:47,260 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 41.66 2021-05-17 22:13:52,371 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 43.69 2021-05-17 22:13:53,903 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,903 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,904 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,957 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:53,957 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3428 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:13:57,467 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 45.59 2021-05-17 22:14:00,496 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,496 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,496 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,497 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,496 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,500 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,500 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,502 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,500 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:00,503 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3631 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:02,611 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 47.66 2021-05-17 22:14:07,739 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 49.29 2021-05-17 22:14:12,832 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 51.14 2021-05-17 22:14:17,923 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 52.22 2021-05-17 22:14:23,517 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 53.75 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,719 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,717 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,719 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:23,719 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4371 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:28,634 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 55.37 2021-05-17 22:14:33,723 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 56.74 2021-05-17 22:14:37,329 | test | INFO | pool-19-thread-4 | [table_view:display:72] Keys failed in bucket2:scope1:FRByk_-35-609000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:38,858 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 58.53 2021-05-17 22:14:43,996 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 60.39 2021-05-17 22:14:48,997 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:48,999 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:48,999 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,000 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:48,999 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,000 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,000 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,000 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,002 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 22:14:49,022 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,023 | test | INFO | pool-19-thread-16 | [table_view:display:72] Keys failed in bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4820 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:14:49,023 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 22:14:49,095 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 62.02 2021-05-17 22:14:54,203 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 63.97 2021-05-17 22:14:59,361 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.96 2021-05-17 22:15:04,480 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 22:15:09,571 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 22:15:12,549 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 22:15:12,549 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 22:15:16,122 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 22:15:21,233 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 68.7 2021-05-17 22:15:25,285 | test | INFO | pool-19-thread-12 | [table_view:display:72] Keys failed in default:_default:mmUlP5LW3xymNW-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-151 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:25,779 | test | INFO | pool-19-thread-4 | [table_view:display:72] Keys failed in default:FoC-mUL85jyETN6z-35-650000:DI8SZmj--terD8XTHr-35-654000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-936 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:26,351 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.39 2021-05-17 22:15:29,042 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in default:_default:M-35-639000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:30,390 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:1tlaEjdl-8AMGAXx-36-983000:Kwj0WyG39-37-152000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2772 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:30,391 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:1tlaEjdl-8AMGAXx-36-983000:Kwj0WyG39-37-152000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2772 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:31,476 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 71.54 2021-05-17 22:15:31,598 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:scope1:D_ekPaVGF4xryJBL-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-678 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:33,555 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:35,930 | test | INFO | pool-19-thread-12 | [table_view:display:72] Keys failed in default:_default:mmUlP5LW3xymNW-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-209 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-399 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-378 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:36,582 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 72.87 2021-05-17 22:15:37,076 | test | INFO | pool-19-thread-2 | [table_view:display:72] Keys failed in default:Wx-35-645000:Gtm14RGJFFpmIERJXMN-35-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-612 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-630 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-663 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:39,572 | test | INFO | pool-19-thread-6 | [table_view:display:72] Keys failed in default:_default:40D7xKXZHlDg%kdhR-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:41,229 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in default:scope2:TMR08-35-643000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-119 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-79 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-63 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-186 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:41,230 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-119 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-79 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-63 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-186 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:41,668 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 74.43 2021-05-17 22:15:46,736 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in default:scope1:0bTEkz-35-641000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-524 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:46,762 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 75.85 2021-05-17 22:15:49,306 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in default:_default:3FyMR-35-635000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-149 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:50,608 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:50,723 | test | INFO | pool-19-thread-13 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-638 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-604 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:51,890 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 77.39 2021-05-17 22:15:54,043 | test | INFO | pool-19-thread-2 | [table_view:display:72] Keys failed in default:Wx-35-645000:Gtm14RGJFFpmIERJXMN-35-646000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-779 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-890 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:55,625 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-328 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-300 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-345 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:56,450 | test | INFO | pool-19-thread-12 | [table_view:display:72] Keys failed in default:_default:mmUlP5LW3xymNW-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-437 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-472 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:15:57,009 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 78.83 2021-05-17 22:15:58,490 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:scope1:D_ekPaVGF4xryJBL-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-842 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:00,019 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:1tlaEjdl-8AMGAXx-36-983000:Kwj0WyG39-37-152000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3019 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:00,019 | test | INFO | pool-19-thread-25 | [table_view:display:72] Keys failed in default:1tlaEjdl-8AMGAXx-36-983000:Kwj0WyG39-37-152000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3019 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3189 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:01,450 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in default:Wx-35-645000:p-38-98000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:01,450 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in default:Wx-35-645000:p-38-98000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-300 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-290 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:02,134 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 80.4 2021-05-17 22:16:03,246 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in default:_default:3FyMR-35-635000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-345 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:03,671 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:06,361 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in default:scope1:0bTEkz-35-641000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:07,270 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 81.72 2021-05-17 22:16:11,489 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-415 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-423 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-443 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-499 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:11,701 | test | INFO | pool-19-thread-2 | [table_view:display:72] Keys failed in default:Wx-35-645000:Gtm14RGJFFpmIERJXMN-35-646000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-919 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:12,059 | test | INFO | pool-19-thread-13 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-816 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-889 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-725 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-766 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-866 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-855 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:12,390 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 83.39 2021-05-17 22:16:16,423 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in default:qRcw-37-333000:WfZqq-37-511000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1173 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:16,424 | test | INFO | pool-19-thread-14 | [table_view:display:72] Keys failed in default:qRcw-37-333000:WfZqq-37-511000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1173 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:17,476 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 84.51 2021-05-17 22:16:17,736 | test | INFO | pool-19-thread-17 | [table_view:display:72] Keys failed in default:_default:3FyMR-35-635000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-406 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-443 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-400 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-483 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-495 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:22,558 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 85.6 2021-05-17 22:16:22,561 | test | INFO | pool-19-thread-6 | [table_view:display:72] Keys failed in default:FoC-mUL85jyETN6z-35-650000:i-InxSS0-35-652000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-513 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-596 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-550 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:22,993 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-515 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-538 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-513 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:23,482 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:scope1:D_ekPaVGF4xryJBL-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-928 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-903 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:24,796 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-87 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-65 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-20 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-112 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-197 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-198 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:24,796 | test | INFO | pool-19-thread-3 | [table_view:display:72] Keys failed in default:_default:Vg1jIQI-35-637000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-87 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-65 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-20 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-112 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-197 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-198 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:27,714 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 87.15 2021-05-17 22:16:29,792 | test | INFO | pool-19-thread-13 | [table_view:display:72] Keys failed in default:scope1:collections_2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-972 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:32,861 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 88.76 2021-05-17 22:16:38,007 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 90.31 2021-05-17 22:16:42,392 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-809 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:42,657 | test | INFO | pool-19-thread-5 | [table_view:display:72] Keys failed in default:scope1:0bTEkz-35-641000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:43,099 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 91.88 2021-05-17 22:16:43,278 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-306 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-296 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:43,831 | test | INFO | pool-19-thread-6 | [table_view:display:72] Keys failed in default:FoC-mUL85jyETN6z-35-650000:i-InxSS0-35-652000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-726 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-710 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-763 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-781 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:44,032 | test | INFO | pool-19-thread-29 | [table_view:display:72] Keys failed in default:_default:i2R-35-639000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-527 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-589 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:48,180 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 93.3 2021-05-17 22:16:51,078 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:40D7xKXZHlDg%kdhR-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-188 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-163 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:53,267 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 94.86 2021-05-17 22:16:54,483 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in default:scope2:TMR08-35-643000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-306 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-327 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-296 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:57,358 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in default:Wx-35-645000:p-38-98000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:57,359 | test | INFO | pool-19-thread-27 | [table_view:display:72] Keys failed in default:Wx-35-645000:p-38-98000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-869 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:16:58,371 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 96.4 2021-05-17 22:17:00,483 | test | INFO | pool-19-thread-20 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-931 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:02,963 | test | INFO | pool-19-thread-18 | [table_view:display:72] Keys failed in default:scope1:t-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-409 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-489 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-462 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:03,469 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 97.98 2021-05-17 22:17:04,967 | test | INFO | pool-19-thread-6 | [table_view:display:72] Keys failed in default:FoC-mUL85jyETN6z-35-650000:i-InxSS0-35-652000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:08,309 | test | INFO | pool-19-thread-29 | [table_view:display:72] Keys failed in default:_default:i2R-35-639000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-828 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-860 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:08,571 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 98.95 2021-05-17 22:17:10,345 | test | INFO | pool-19-thread-30 | [table_view:display:72] Keys failed in default:scope2:TMR08-35-643000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-403 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-490 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:13,239 | test | INFO | pool-19-thread-24 | [table_view:display:72] Keys failed in default:scope1:Ot-35-642000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-118 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-92 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-144 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-123 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:13,673 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 22:17:18,759 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 22:17:21,171 | test | INFO | pool-19-thread-23 | [table_view:display:72] Keys failed in default:_default:40D7xKXZHlDg%kdhR-35-638000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:21,171 | test | INFO | pool-19-thread-3 | [table_view:display:72] Keys failed in default:_default:Vg1jIQI-35-637000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-359 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-364 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:21,247 | test | INFO | pool-19-thread-12 | [table_view:display:72] Keys failed in default:scope1:yFZ-msA0wgrnFQw8_h-35-640000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-559 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:21,539 | test | INFO | pool-19-thread-29 | [table_view:display:72] Keys failed in default:_default:i2R-35-639000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:17:23,938 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 22:17:34,233 | test | INFO | pool-19-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 22:17:34,404 | test | INFO | pool-19-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 365.648000002 sec 2021-05-17 22:19:35,124 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 22:19:35,125 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 22:19:35,128 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-17 22:19:35,128 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 22:20:09,832 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 22:20:09,841 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-17 22:20:09,842 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 22:21:34,506 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 22:21:34,506 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 22:21:34,509 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 3 docs from 500 to 980 2021-05-17 22:21:34,510 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 14 docs from 0 to 500 2021-05-17 22:21:34,512 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-17 22:21:34,513 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 12 docs from 500 to 980 2021-05-17 22:21:34,515 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-17 22:21:34,516 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 17 docs from 0 to 500 2021-05-17 22:21:34,517 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 17 docs from 0 to 500 2021-05-17 22:22:32,588 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-17 22:22:32,591 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 12 docs from 500 to 980 2021-05-17 22:22:32,592 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 15 docs from 500 to 980 2021-05-17 22:22:32,594 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 10 docs from 0 to 500 2021-05-17 22:22:32,596 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 15 docs from 0 to 500 2021-05-17 22:22:32,596 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 12 docs from 500 to 980 2021-05-17 22:22:32,598 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-17 22:22:32,598 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 18 docs from 0 to 500 2021-05-17 22:22:32,598 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-17 22:22:32,599 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 8 docs from 0 to 500 2021-05-17 22:22:32,601 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 8 docs from 0 to 500 2021-05-17 22:22:32,601 | test | ERROR | pool-19-thread-7 | [task:call:3809] Failed to load 6 docs from 500 to 980 2021-05-17 22:29:06,904 | test | INFO | pool-19-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***................................................................................................***X | 724.155 | | 229.257 | .............................X | 229.257 | | 56.918 | ......***...............................................................*....................**X | 678.666 | | 288.435 | .....................................***X | 288.435 | | 84.787 | ..........***X | 84.787 | | 65.008 | .......X | 65.008 | | 62.298 | .......***..............*..............................................*X | 516.691 | | 516.691 | ...................................................................*X | 516.691 | | 516.593 | ...................................................................*.....***X | 554.215 | | 94.316 | ...........****............................*....................***X | 455.754 | | 455.754 | ...........................................................*.......****.................****X | 631.633 | | 180.971 | .......................****..X | 200.672 | | 200.622 | .........................**X | 200.622 | | 127.512 | ................*****X | 129.093 | | 126.375 | ...............***........................................*...........***X | 508.964 | | 477.732 | ..............................................................***X | 477.732 | | 210.916 | ...........................***X | 210.916 | | 142.081 | .................*****........*****......................................*....................................***X | 749.793 | | 402.054 | ....................................................*****.................***X | 526.314 | | 472.587 | ..............................................................X | 472.587 | | 445.692 | ..........................................................***X | 445.692 | | 275.218 | ...................................****X | 275.218 | | 180.295 | .......................*****..*****.***X | 205.903 | | 139.889 | .................****X | 139.889 | | 126.430 | ...............****..............***X | 232.237 | | 84.010 | ..........*****...........X | 168.155 | | 168.155 | .....................**X | 168.155 | | 168.155 | .....................*.......................****X | 344.787 | | 344.787 | ............................................X | 344.787 | | 340.368 | ............................................*****X | 340.368 | | 247.428 | ...............................X | 247.428 | | 247.428 | ...............................****X | 247.428 | | 202.995 | ..........................******...........................******...*....****X | 464.269 | | 424.469 | .......................................................X | 424.469 | | 354.876 | ..............................................*****X | 354.876 | | 301.395 | .......................................*****X | 301.395 | | 230.965 | .............................*****X | 230.965 | | 224.841 | ............................******...******X | 243.668 | | 224.907 | ............................******......................***X | 386.531 | | 386.531 | ..................................................**...X | 405.619 | | 402.479 | ....................................................*X | 402.479 | | 402.479 | ....................................................*X | 402.479 | | 402.479 | ....................................................*.*.......*X | 462.430 | | 462.430 | ............................................................***.X | 469.416 | | 454.154 | ...........................................................*****...........*****X | 537.061 | | 398.709 | ....................................................******............X | 491.263 | | 491.263 | ................................................................**X | 491.263 | | 491.263 | ................................................................*X | 491.263 | | 415.914 | ......................................................******.*..****X | 437.606 | | 409.630 | .....................................................*X | 409.630 | | 409.630 | .....................................................****......*****X | 452.709 | | 408.781 | .....................................................***********X | 410.975 | | 396.051 | ...................................................X | 396.051 | | 391.663 | ...................................................****X | 391.663 | | 371.900 | ................................................*******.............*****X | 469.229 | | 383.460 | ..................................................******.....***X | 425.600 | | 425.600 | .......................................................*X | 425.600 | | 352.500 | ..............................................******........*X | 418.409 | | 418.409 | ......................................................***X | 418.409 | | 406.445 | .....................................................******...*****X | 430.602 | | 390.599 | ...................................................*****X | 390.599 | | 344.076 | ............................................******........*****X | 402.406 | | 370.182 | ................................................******..*X | 385.948 | | 385.948 | ..................................................***X | 385.948 | | 297.107 | ......................................******.....******.....*****X | 369.339 | | 327.494 | ..........................................******...******...*****X | 374.175 | | 343.688 | ............................................*******..****X | 358.200 | | 335.509 | ...........................................X | 335.509 | | 312.576 | ........................................******.......******..*****X | 379.900 | | 344.308 | ............................................******...******........*****X | 422.056 | | 360.253 | ...............................................*****X | 360.253 | | 345.910 | .............................................*******........*****X | 411.012 | | 331.463 | ...........................................*****X | 331.463 | | 316.688 | .........................................******..*......*****..******..........*****X | 467.157 | | 443.473 | ..........................................................X | 443.473 | | 405.673 | .....................................................****X | 405.673 | | 398.286 | ....................................................*****X | 398.286 | | 361.351 | ...............................................******...******.*......******.......******.....************............*****X | 616.156 | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 22:29:08,207 | test | INFO | pool-19-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.****.............*..***..*.......**.*...**X | 94.082 | | 92.371 | ...........................*.......****...*..........**X | 153.889 | | 123.482 | .....................................X | 123.482 | | 99.593 | ..............................****...X | 110.979 | | 106.403 | ................................***.......*.....................*****.*****......................................***X | 319.884 | | 185.782 | .........................................................****X | 185.782 | | 138.087 | ..........................................***X | 138.087 | | 112.681 | ..................................*....**X | 126.527 | | 125.235 | ......................................****X | 125.235 | | 115.115 | ..................................*..***............*....***X | 171.238 | | 120.026 | ....................................***X | 120.519 | | 105.792 | ................................***X | 105.792 | | 21.198 | .....***X | 21.198 | | 19.198 | .....X | 19.198 | | 1.300 | *********X | 2.700 | | 1.000 | ********X | 2.000 | | 1.599 | ****X | 1.599 | | 0.400 | ********X | 2.100 | | 0.700 | ****X | 0.700 | | 0.600 | ***X | 0.600 | | 0.200 | ********X | 0.800 | | 0.800 | ********X | 2.902 | | 2.702 | X | 2.702 | | 0.400 | ***X | 0.400 | | 0.000 | ************************X | 3.100 | | 1.000 | ************............******...........*****X | 73.799 | | 72.099 | .....................X | 72.099 | | 67.689 | ....................************.*****X | 72.091 | | 69.602 | ....................*****X | 69.602 | | 67.904 | ....................X | 67.904 | | 60.088 | .................****X | 60.088 | | 53.710 | ...............*****X | 53.710 | | 0.800 | ************************X | 3.000 | | 0.000 | *****************X | 2.301 | | 0.900 | ***********X | 2.799 | | 2.100 | X | 2.100 | | 2.000 | ****X | 2.000 | | 1.600 | ***********X | 2.100 | | 0.400 | *****************X | 2.699 | | 1.700 | ***X | 1.700 | | 1.700 | *******X | 2.200 | | 1.900 | *****X | 1.900 | | 1.700 | X | 1.700 | | 1.100 | **********X | 1.300 | | 1.200 | X | 1.200 | | 1.000 | ***********X | 2.100 | | 1.300 | *****X | 1.300 | | 1.000 | ******..*****X | 6.699 | | 1.600 | *****X | 1.600 | | 0.300 | ******.*****X | 4.197 | | 0.700 | ***********X | 1.900 | | 1.800 | ****X | 1.800 | | 1.000 | X | 1.000 | | 0.700 | **********X | 1.900 | | 1.300 | X | 1.300 | | 0.700 | ***********X | 2.500 | | 1.200 | ******.*****X | 5.700 | | 4.900 | X | 4.900 | | 2.400 | ************.*****X | 3.400 | | 1.100 | ***********X | 2.300 | | 0.000 | ***********X | 2.200 | | 1.300 | *****X | 1.300 | | 1.000 | *******.*****X | 3.899 | | 0.600 | ******.*****X | 3.900 | | 1.500 | ******X | 1.700 | | 1.400 | ****X | 1.400 | | 0.400 | ******.*****X | 5.500 | | 2.900 | ******.*****X | 3.600 | | 1.200 | *****X | 1.200 | | 0.000 | X | 0.000 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 22:29:09,536 | test | INFO | pool-19-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................***X | 489.511 | | 45.305 | ........***X | 45.305 | | 18.595 | ..*....*X | 35.800 | | 29.500 | .....X | 29.500 | | 18.100 | ..****..............***X | 87.403 | | 86.315 | ................X | 86.315 | | 84.600 | ................**X | 84.600 | | 49.418 | .........X | 49.418 | | 31.511 | .....****......*....***X | 81.895 | | 80.106 | ...............X | 80.106 | | 71.718 | .............**X | 71.718 | | 49.207 | .........*.***X | 56.999 | | 54.899 | ..........*.............................***X | 200.547 | | 172.432 | ..................................X | 172.432 | | 153.484 | ..............................***X | 153.484 | | 145.464 | ............................****X | 146.458 | | 142.912 | ............................**X | 142.912 | | 125.512 | ........................*.....***X | 148.392 | | 143.872 | ............................X | 143.872 | | 123.572 | ........................***.............*.***X | 192.953 | | 185.442 | ....................................***X | 185.442 | | 122.290 | .......................****............*****...*****......*..***X | 231.750 | | 43.097 | .......****....****X | 61.622 | | 50.168 | .........********X | 53.437 | | 50.300 | .........******.****...*...**X | 87.603 | | 59.997 | ...........****X | 59.997 | | 52.612 | .........****.***X | 57.481 | | 53.570 | .........X | 53.570 | | 45.710 | ........****......*****X | 73.524 | | 68.498 | ............******.X | 72.807 | | 67.814 | ............*****....*****X | 86.309 | | 67.799 | ............*****X | 67.799 | | 19.303 | ..*****X | 19.303 | | 0.000 | *****************X | 0.200 | | 0.000 | ************X | 0.400 | | 0.400 | ***********X | 2.800 | | 2.400 | X | 2.400 | | 0.500 | ****************X | 1.700 | | 0.400 | *****X | 0.400 | | 0.000 | ***********X | 3.500 | | 0.500 | ***********X | 2.500 | | 1.501 | X | 1.501 | | 0.500 | *****X | 0.500 | | 0.000 | *****************X | 2.400 | | 1.600 | ***********X | 2.400 | | 0.400 | *****X | 0.400 | | 0.000 | ***********************X | 2.400 | | 1.300 | *****X | 1.300 | | 0.700 | *****X | 0.700 | | 0.000 | ***********************X | 2.400 | | 2.100 | X | 2.100 | | 1.400 | ***********X | 2.400 | | 0.400 | *****X | 0.400 | | 0.300 | ***********X | 1.199 | | 0.600 | ***********X | 2.100 | | 0.500 | *****X | 0.500 | | 0.000 | ******************X | 2.100 | | 1.200 | ****X | 1.200 | | 0.900 | *****X | 0.900 | | 0.300 | X | 0.300 | | 0.000 | *****************X | 2.601 | | 0.600 | *****X | 0.600 | | 0.200 | ***********X | 1.300 | | 0.800 | ***********X | 1.300 | | 0.000 | ******************X | 3.700 | | 0.600 | **********X | 1.300 | | 1.300 | *****X | 1.300 | | 0.500 | X | 0.500 | | 0.300 | ****************X | 1.400 | | 0.500 | ***********X | 1.800 | | 1.600 | *****X | 1.600 | | 0.000 | *X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 22:29:09,582 | test | WARNING | Thread-42767 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Vg1jIQI-35-637000 delete failed keys from task: [u'test_collections-389', u'test_collections-364', u'test_collections-321', u'test_collections-286', u'test_collections-87', u'test_collections-65', u'test_collections-219', u'test_collections-359', u'test_collections-316', u'test_collections-86', u'test_collections-20', u'test_collections-112', u'test_collections-197', u'test_collections-198', u'test_collections-371'] 2021-05-17 22:29:09,586 | test | WARNING | Thread-42769 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:3FyMR-35-635000 delete failed keys from task: [u'test_collections-406', u'test_collections-149', u'test_collections-97', u'test_collections-322', u'test_collections-443', u'test_collections-345', u'test_collections-400', u'test_collections-232', u'test_collections-483', u'test_collections-495'] 2021-05-17 22:29:09,589 | test | WARNING | Thread-42777 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:mmUlP5LW3xymNW-35-638000 delete failed keys from task: [u'test_collections-209', u'test_collections-428', u'test_collections-468', u'test_collections-125', u'test_collections-1', u'test_collections-437', u'test_collections-399', u'test_collections-114', u'test_collections-378', u'test_collections-472', u'test_collections-450', u'test_collections-151', u'test_collections-152', u'test_collections-270'] 2021-05-17 22:29:09,591 | test | WARNING | Thread-42803 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:1tlaEjdl-8AMGAXx-36-983000:Kwj0WyG39-37-152000 create failed keys from task: [u'test_collections-3019', u'test_collections-2772', u'test_collections-3189'] 2021-05-17 22:29:09,589 | test | WARNING | Thread-42794 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:40D7xKXZHlDg%kdhR-35-638000 update failed keys from task: [u'test_collections-906', u'test_collections-943', u'test_collections-971'] 2021-05-17 22:29:09,604 | test | WARNING | Thread-42829 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 update failed keys from task: [u'test_collections-506', u'test_collections-528', u'test_collections-888', u'test_collections-962', u'test_collections-834', u'test_collections-896', u'test_collections-798', u'test_collections-675', u'test_collections-950', u'test_collections-694', u'test_collections-871'] 2021-05-17 22:29:09,612 | test | WARNING | Thread-42787 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:M-35-639000 update failed keys from task: [u'test_collections-938', u'test_collections-953'] 2021-05-17 22:29:09,638 | test | WARNING | Thread-42899 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:TMR08-35-643000 delete failed keys from task: [u'test_collections-306', u'test_collections-79', u'test_collections-403', u'test_collections-327', u'test_collections-446', u'test_collections-289', u'test_collections-386', u'test_collections-186', u'test_collections-160', u'test_collections-319', u'test_collections-119', u'test_collections-438', u'test_collections-216', u'test_collections-63', u'test_collections-296', u'test_collections-9', u'test_collections-490'] 2021-05-17 22:29:09,651 | test | WARNING | Thread-42956 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:5_1-5yKrKiznWIhZiFX-34-806000 create failed keys from task: [u'test_collections-3391', u'test_collections-4371', u'test_collections-3428', u'test_collections-3201', u'test_collections-4820', u'test_collections-3323', u'test_collections-3631'] 2021-05-17 22:29:09,655 | test | WARNING | Thread-42794 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:40D7xKXZHlDg%kdhR-35-638000 delete failed keys from task: [u'test_collections-69', u'test_collections-76', u'test_collections-219', u'test_collections-334', u'test_collections-389', u'test_collections-188', u'test_collections-371', u'test_collections-163'] 2021-05-17 22:29:09,664 | test | WARNING | Thread-42995 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:muIRICWuf-33-564000 create failed keys from task: [u'test_collections-3391', u'test_collections-4371', u'test_collections-3428', u'test_collections-3201', u'test_collections-4820', u'test_collections-3323', u'test_collections-3631'] 2021-05-17 22:29:09,664 | test | WARNING | Thread-42996 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:j%jPTwwNsQ5B%j-fLq-32-802000:qay8CWLIWsxEL-33-564000 create failed keys from task: [u'test_collections-3391', u'test_collections-4371', u'test_collections-3428', u'test_collections-3201', u'test_collections-4820', u'test_collections-3323', u'test_collections-3631'] 2021-05-17 22:29:09,676 | test | WARNING | Thread-43039 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope1:i4MQdW_-35-609000 delete failed keys from task: [u'test_collections-1'] 2021-05-17 22:29:09,651 | test | WARNING | Thread-42957 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:StbXoZ4aXSKuaeVr5_Y-34-330000:zT%WKFsLu-34-805000 create failed keys from task: [u'test_collections-3391', u'test_collections-4371', u'test_collections-3428', u'test_collections-3201', u'test_collections-4820', u'test_collections-3323', u'test_collections-3631'] 2021-05-17 22:29:09,750 | test | WARNING | Thread-42834 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Wx-35-645000:Gtm14RGJFFpmIERJXMN-35-646000 update failed keys from task: [u'test_collections-927', u'test_collections-749', u'test_collections-502', u'test_collections-744', u'test_collections-663', u'test_collections-919', u'test_collections-915', u'test_collections-779', u'test_collections-612', u'test_collections-630', u'test_collections-794', u'test_collections-890'] 2021-05-17 22:29:09,753 | test | WARNING | Thread-43035 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:bB6fBRIQ6p-33-915000:o3bI9-T%fIaen-34-55000 create failed keys from task: [u'test_collections-3391', u'test_collections-4371', u'test_collections-3428', u'test_collections-3201', u'test_collections-4820', u'test_collections-3323', u'test_collections-3631'] 2021-05-17 22:29:09,760 | test | WARNING | Thread-42852 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Ot-35-642000 delete failed keys from task: [u'test_collections-118', u'test_collections-92', u'test_collections-133', u'test_collections-144', u'test_collections-166', u'test_collections-101', u'test_collections-123', u'test_collections-7'] 2021-05-17 22:29:09,762 | test | WARNING | Thread-43046 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope1:FRByk_-35-609000 update failed keys from task: [u'test_collections-4'] 2021-05-17 22:29:09,767 | test | WARNING | Thread-43226 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:bTWZ-kK%kNkNzU_f-35-622000 delete failed keys from task: [u'test_collections-1'] 2021-05-17 22:29:09,782 | test | WARNING | Thread-43231 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:CxnncA4-zfnO-35-625000 delete failed keys from task: [u'test_collections-1'] 2021-05-17 22:29:09,812 | test | WARNING | Thread-42844 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collections_2 update failed keys from task: [u'test_collections-805', u'test_collections-889', u'test_collections-604', u'test_collections-725', u'test_collections-766', u'test_collections-866', u'test_collections-921', u'test_collections-540', u'test_collections-760', u'test_collections-816', u'test_collections-937', u'test_collections-638', u'test_collections-810', u'test_collections-855', u'test_collections-972'] 2021-05-17 22:29:09,835 | test | WARNING | Thread-42830 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Wx-35-645000:p-38-98000 create failed keys from task: [u'test_collections-869', u'test_collections-300', u'test_collections-290'] 2021-05-17 22:29:09,838 | test | WARNING | Thread-42856 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:D_ekPaVGF4xryJBL-35-642000 update failed keys from task: [u'test_collections-928', u'test_collections-506', u'test_collections-528', u'test_collections-903', u'test_collections-678', u'test_collections-744', u'test_collections-842', u'test_collections-543', u'test_collections-694', u'test_collections-794'] 2021-05-17 22:29:09,854 | test | WARNING | Thread-42800 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:i2R-35-639000 update failed keys from task: [u'test_collections-828', u'test_collections-527', u'test_collections-964', u'test_collections-589', u'test_collections-784', u'test_collections-860', u'test_collections-880', u'test_collections-735', u'test_collections-711', u'test_collections-611', u'test_collections-754', u'test_collections-697', u'test_collections-670'] 2021-05-17 22:29:09,869 | test | WARNING | Thread-42847 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:0bTEkz-35-641000 update failed keys from task: [u'test_collections-827', u'test_collections-805', u'test_collections-608', u'test_collections-729', u'test_collections-803', u'test_collections-846', u'test_collections-524', u'test_collections-921', u'test_collections-840', u'test_collections-917', u'test_collections-937', u'test_collections-555'] 2021-05-17 22:29:09,907 | test | WARNING | Thread-42862 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:yFZ-msA0wgrnFQw8_h-35-640000 update failed keys from task: [u'test_collections-614', u'test_collections-559', u'test_collections-549', u'test_collections-586', u'test_collections-692', u'test_collections-593'] 2021-05-17 22:29:09,936 | test | WARNING | Thread-42871 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:t-35-642000 update failed keys from task: [u'test_collections-809', u'test_collections-909', u'test_collections-707', u'test_collections-726', u'test_collections-763', u'test_collections-660', u'test_collections-781', u'test_collections-515', u'test_collections-538', u'test_collections-513', u'test_collections-832', u'test_collections-931'] 2021-05-17 22:29:09,966 | test | WARNING | Thread-42944 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:FoC-mUL85jyETN6z-35-650000:DI8SZmj--terD8XTHr-35-654000 update failed keys from task: [u'test_collections-936'] 2021-05-17 22:29:10,025 | test | WARNING | Thread-42952 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:FoC-mUL85jyETN6z-35-650000:i-InxSS0-35-652000 update failed keys from task: [u'test_collections-707', u'test_collections-726', u'test_collections-967', u'test_collections-763', u'test_collections-660', u'test_collections-781', u'test_collections-955', u'test_collections-513', u'test_collections-910', u'test_collections-556', u'test_collections-710', u'test_collections-596', u'test_collections-550'] 2021-05-17 22:29:10,072 | test | WARNING | Thread-42955 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:qRcw-37-333000:WfZqq-37-511000 create failed keys from task: [u'test_collections-1173'] 2021-05-17 22:29:10,285 | test | WARNING | Thread-42829 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Wx-35-645000:miY5X%lWwXjsyQ9-35-646000 delete failed keys from task: [u'test_collections-328', u'test_collections-79', u'test_collections-423', u'test_collections-300', u'test_collections-322', u'test_collections-443', u'test_collections-345', u'test_collections-186', u'test_collections-160', u'test_collections-119', u'test_collections-415', u'test_collections-63', u'test_collections-499', u'test_collections-232', u'test_collections-495', u'test_collections-9', u'test_collections-290'] 2021-05-17 22:29:10,733 | test | WARNING | Thread-42871 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:t-35-642000 delete failed keys from task: [u'test_collections-409', u'test_collections-306', u'test_collections-327', u'test_collections-489', u'test_collections-289', u'test_collections-462', u'test_collections-87', u'test_collections-65', u'test_collections-319', u'test_collections-86', u'test_collections-20', u'test_collections-259', u'test_collections-278', u'test_collections-112', u'test_collections-255', u'test_collections-197', u'test_collections-296', u'test_collections-198'] 2021-05-17 22:29:19,677 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 22:30:10,940 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 22:30:10,940 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 22:30:34,602 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 97% 2021-05-17 22:30:34,604 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 97% 2021-05-17 22:30:48,703 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 22:30:48,704 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 22:31:54,401 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1500000954 seconds to complete 2021-05-17 22:31:54,417 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2681.257s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 17 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_17 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_18 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_18', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 18, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 22:32:03,812 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #18 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 22:32:04,523 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #18 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 22:32:04,947 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x572b57 type=client open_count=1 channel=[id: 0x64dec300, 0.0.0.0/0.0.0.0:46213] timeout=300.0>) 2021-05-17 22:32:25,842 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x572b58 type=client open_count=1 channel=[id: 0x675c00ca, 0.0.0.0/0.0.0.0:36985] timeout=300.0>) 2021-05-17 22:32:27,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x572b59 type=client open_count=1 channel=[id: 0xd0fc593a, 0.0.0.0/0.0.0.0:44409] timeout=300.0>) 2021-05-17 22:32:29,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x572b5a type=client open_count=1 channel=[id: 0xb133844f, 0.0.0.0/0.0.0.0:53829] timeout=300.0>) 2021-05-17 22:32:53,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x572b5b type=client open_count=1 channel=[id: 0xa7ef03a7, 0.0.0.0/0.0.0.0:34649] timeout=300.0>) 2021-05-17 22:32:55,882 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x572b5c type=client open_count=1 channel=[id: 0x72e3f81a, 0.0.0.0/0.0.0.0:33873] timeout=300.0>) 2021-05-17 22:32:57,908 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x572b5d type=client open_count=1 channel=[id: 0xff4214cd, 0.0.0.0/0.0.0.0:35586] timeout=300.0>) 2021-05-17 22:33:22,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x572b5e type=client open_count=1 channel=[id: 0x8db34e80, 0.0.0.0/0.0.0.0:51874] timeout=300.0>) 2021-05-17 22:33:24,701 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x572b5f type=client open_count=1 channel=[id: 0xf8216e57, 0.0.0.0/0.0.0.0:51131] timeout=300.0>) 2021-05-17 22:33:26,727 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x572b60 type=client open_count=1 channel=[id: 0xcd8c2c6e, 0.0.0.0/0.0.0.0:49813] timeout=300.0>) 2021-05-17 22:33:51,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x572b61 type=client open_count=1 channel=[id: 0x4cece6d0, 0.0.0.0/0.0.0.0:34455] timeout=300.0>) 2021-05-17 22:33:53,085 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x572b62 type=client open_count=1 channel=[id: 0x6bb1de36, 0.0.0.0/0.0.0.0:47054] timeout=300.0>) 2021-05-17 22:33:55,111 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x572b63 type=client open_count=1 channel=[id: 0x79756a67, 0.0.0.0/0.0.0.0:60698] timeout=300.0>) 2021-05-17 22:34:19,141 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x572b64 type=client open_count=1 channel=[id: 0x0f902b57, 0.0.0.0/0.0.0.0:59465] timeout=300.0>) 2021-05-17 22:34:21,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x572b65 type=client open_count=1 channel=[id: 0xf963853d, 0.0.0.0/0.0.0.0:53236] timeout=300.0>) 2021-05-17 22:34:23,183 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 22:34:50,700 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 22:35:04,592 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 22:35:04,615 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #18 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 22:35:04,660 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #18 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 22:35:14,905 | test | INFO | pool-20-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 1.745 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+-------+--------------+ 2021-05-17 22:35:14,930 | test | INFO | pool-20-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:35:25,003 | test | INFO | pool-20-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 22:35:25,028 | test | INFO | pool-20-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.123000145 sec 2021-05-17 22:35:25,288 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 22:36:21,184 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 22:36:21,535 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 668200608 | 83464476 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 680596304 | 101403433 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 722967824 | 65471668 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 22:36:21,536 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 22:37:00,930 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 22:37:00,940 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 22:55:13,076 | test | INFO | pool-20-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***........................................................................................*****.......*.....****X | 72.532 | | 13.172 | .................X | 13.172 | | 0.000 | *****************************************.*...*****..*.......****X | 9.802 | | 8.199 | ..........X | 8.199 | | 1.901 | .*****...*............****X | 13.001 | | 12.000 | ...............X | 12.000 | | 7.602 | .........****X | 7.602 | | 5.899 | .......X | 5.899 | | 0.401 | ****X | 0.401 | | 0.000 | ******************************....*..****X | 4.500 | | 0.899 | X | 0.899 | | 0.000 | *****************....*....****X | 6.001 | | 2.799 | ..X | 2.799 | | 0.000 | *****.*.****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | ***********************************....****X | 3.000 | | 2.199 | ..X | 2.199 | | 0.000 | ******..******......*****X | 6.300 | | 5.700 | ......*****X | 5.700 | | 0.000 | ******....*****X | 3.000 | | 1.400 | X | 1.400 | | 0.300 | *****.******.*..****X | 3.001 | | 1.400 | X | 1.400 | | 0.000 | *****..*****X | 1.500 | | 0.000 | ******..*****X | 1.500 | | 0.000 | ******......*****X | 4.501 | | 1.500 | .****X | 1.500 | | 1.500 | .*..*****X | 3.600 | | 2.700 | ..****X | 2.700 | | 1.900 | .X | 1.900 | | 1.200 | ****X | 1.200 | | 0.800 | X | 0.800 | | 0.000 | *****.*.....****X | 4.501 | | 3.300 | ...X | 3.300 | | 0.000 | ******.*.****..*..***X | 4.500 | | 4.100 | ....X | 4.100 | | 3.000 | ...****X | 3.000 | | 2.200 | ..X | 2.200 | | 0.000 | ******..*****.*.****X | 3.000 | | 0.000 | ******..******......****X | 6.001 | | 5.201 | ......X | 5.201 | | 4.401 | .....X | 4.401 | | 3.000 | ...**********X | 3.001 | | 1.500 | .****X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | *****************.*.****X | 1.500 | | 1.100 | X | 1.100 | | 1.100 | *****..*.****X | 3.401 | | 2.200 | ..X | 2.200 | | 1.499 | .****X | 1.499 | | 0.300 | X | 0.300 | | 0.000 | ***********..*****.*****X | 2.600 | | 2.200 | ..X | 2.200 | | 1.500 | .****X | 1.500 | | 1.100 | X | 1.100 | | 0.700 | X | 0.700 | | 0.000 | ****.*.....*****X | 4.501 | | 0.000 | ***********.*.....****X | 4.500 | | 3.300 | ...X | 3.300 | | 0.000 | *****.*.****X | 1.500 | | 0.000 | ************...*.*****..X | 4.400 | | 3.900 | ....***X | 3.900 | | 2.700 | ..X | 2.700 | | 1.500 | .X | 1.500 | | 0.000 | *****.*.****X | 1.500 | | 1.500 | .X | 1.500 | | 1.499 | .****X | 1.499 | | 0.300 | X | 0.300 | | 0.000 | *****************.....*.....***X | 7.498 | | 5.498 | ......X | 5.498 | | 0.000 | **************************************************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 22:55:13,548 | test | INFO | pool-20-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***...................................*****..................................*...................*............***X | 99.441 | | 46.560 | .............................................X | 46.560 | | 0.000 | ************************************************........*..****X | 10.202 | | 9.401 | ........X | 9.401 | | 9.002 | ........*****..*****X | 11.601 | | 7.599 | ......X | 7.599 | | 3.401 | ..****X | 3.401 | | 2.200 | .X | 2.200 | | 1.500 | *****.....*..****X | 8.498 | | 4.097 | ...X | 4.097 | | 2.001 | .****X | 2.001 | | 0.000 | ******************......*****X | 6.000 | | 0.000 | ******.*..****X | 3.000 | | 1.500 | ******..*.****X | 4.501 | | 2.099 | .X | 2.099 | | 0.000 | ***********.****X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ******************************.****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | *****.*..*****.*...******.......****X | 14.700 | | 10.298 | .........X | 10.298 | | 0.000 | ******...*****X | 3.000 | | 1.000 | *******.****X | 2.000 | | 1.600 | X | 1.600 | | 1.501 | ****X | 1.501 | | 0.700 | X | 0.700 | | 0.000 | *****************.****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | ******.*****.*.******..****X | 5.101 | | 4.701 | ...X | 4.701 | | 2.700 | .****X | 2.700 | | 2.300 | .X | 2.300 | | 1.500 | X | 1.500 | | 1.200 | ***X | 1.200 | | 1.200 | ******.*....****X | 6.001 | | 4.400 | ...X | 4.400 | | 2.800 | .X | 2.800 | | 0.000 | *****.*****X | 1.500 | | 1.500 | *****.....****X | 6.001 | | 4.400 | ...X | 4.400 | | 0.000 | *****************************.****X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | ****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | *****.*..****X | 3.000 | | 2.200 | .X | 2.200 | | 1.500 | ****X | 1.500 | | 0.000 | ******************...****X | 3.000 | | 2.600 | .X | 2.600 | | 1.500 | ****X | 1.500 | | 1.100 | X | 1.100 | | 0.700 | X | 0.700 | | 0.000 | *****..*.***X | 3.001 | | 2.200 | .X | 2.200 | | 0.600 | X | 0.600 | | 0.000 | *****.*..***X | 3.000 | | 2.200 | .X | 2.200 | | 1.500 | ****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | ************...****X | 3.001 | | 1.400 | X | 1.400 | | 0.000 | ***********..*..*****X | 4.500 | | 0.000 | *****.******...****X | 4.500 | | 3.300 | ..X | 3.300 | | 0.900 | X | 0.900 | | 0.000 | ******************************************************.***X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | ***********************.****X | 1.500 | | 1.100 | X | 1.100 | | 0.000 | *********************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 22:55:14,667 | test | INFO | pool-20-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***......*****.........................................*..............................................*****.......X | 698.581 | | 678.008 | ................................................................................................****X | 678.008 | | 600.473 | ....................................................................................X | 600.473 | | 584.708 | ..................................................................................****X | 584.708 | | 526.188 | ..........................................................................X | 526.188 | | 500.992 | ......................................................................*****..........*.*****.......*.......****X | 677.219 | | 616.099 | .......................................................................................X | 616.099 | | 528.795 | ..........................................................................****X | 528.795 | | 522.831 | .........................................................................*...****X | 543.193 | | 522.330 | .........................................................................X | 522.330 | | 480.910 | ...................................................................****X | 480.910 | | 449.814 | ...............................................................X | 449.814 | | 412.130 | .........................................................*****...*....****X | 458.342 | | 456.108 | ................................................................X | 456.108 | | 444.312 | ..............................................................*****.X | 454.047 | | 447.855 | ...............................................................*****.........*****X | 515.725 | | 506.077 | .......................................................................****X | 506.077 | | 497.989 | ......................................................................*.****X | 506.690 | | 500.794 | ......................................................................X | 500.794 | | 498.632 | ......................................................................****X | 498.632 | | 481.704 | ...................................................................X | 481.704 | | 444.921 | ..............................................................*****.....*........****X | 535.201 | | 522.345 | .........................................................................*...*****....X | 572.088 | | 542.725 | ............................................................................****X | 542.725 | | 531.907 | ...........................................................................X | 531.907 | | 531.170 | ...........................................................................****X | 531.170 | | 524.835 | ..........................................................................X | 524.835 | | 450.938 | ...............................................................****X | 450.938 | | 450.840 | ...............................................................X | 450.840 | | 412.281 | ..........................................................*****...........*****X | 494.443 | | 459.640 | ................................................................X | 459.640 | | 427.193 | ............................................................*****......****X | 473.002 | | 461.281 | .................................................................*..****X | 480.391 | | 427.854 | ............................................................X | 427.854 | | 408.370 | .........................................................*****............******...*****.X | 519.671 | | 490.688 | .....................................................................****X | 490.688 | | 476.388 | ...................................................................X | 476.388 | | 458.266 | ................................................................****X | 458.266 | | 442.514 | ..............................................................X | 442.514 | | 374.641 | ....................................................*****....*.......****X | 449.698 | | 418.346 | ..........................................................X | 418.346 | | 394.068 | .......................................................*****..*..*****...*.*****..X | 465.849 | | 413.887 | ..........................................................****X | 413.887 | | 406.487 | .........................................................*....*****X | 436.468 | | 367.409 | ...................................................*****...*...........*****...*.......****X | 536.437 | | 522.037 | .........................................................................X | 522.037 | | 458.724 | ................................................................****X | 458.724 | | 416.224 | ..........................................................X | 416.224 | | 346.004 | ................................................*****.....*.............*****..X | 485.367 | | 456.875 | ................................................................****X | 456.875 | | 452.199 | ...............................................................******.....*****X | 488.461 | | 443.340 | ..............................................................****X | 443.340 | | 428.147 | ............................................................X | 428.147 | | 380.455 | .....................................................*****...*.......*.........***X | 511.364 | | 502.726 | ......................................................................X | 502.726 | | 481.294 | ...................................................................X | 481.294 | | 443.322 | ..............................................................****X | 443.322 | | 441.225 | ..............................................................X | 441.225 | | 439.839 | .............................................................*****..****X | 452.810 | | 431.567 | ............................................................X | 431.567 | | 431.478 | ............................................................*..****......*..****X | 502.447 | | 437.180 | .............................................................X | 437.180 | | 378.132 | .....................................................*****.******..........*****.......X | 509.316 | | 480.528 | ...................................................................****X | 480.528 | | 462.314 | .................................................................*****X | 464.658 | | 427.834 | ............................................................X | 427.834 | | 426.177 | ............................................................*****X | 429.598 | | 407.558 | .........................................................*****.....*......****X | 485.360 | | 476.369 | ...................................................................X | 476.369 | | 443.968 | ..............................................................****X | 443.968 | | 366.801 | ...................................................*****..*.*....****..*...*..***X | 467.811 | | 456.910 | ................................................................X | 456.910 | | 444.089 | ..............................................................X | 444.089 | | 413.453 | ..........................................................***X | 413.453 | | 412.242 | ..........................................................X | 412.242 | | 397.590 | .......................................................*...***X | 417.513 | | 416.513 | ..........................................................*....****X | 441.887 | | 432.815 | ............................................................X | 432.815 | | 379.947 | .....................................................*****....*........****X | 464.526 | | 429.778 | ............................................................X | 429.778 | | 404.641 | ........................................................****X | 404.641 | | 372.268 | ....................................................*.*****.....*...*****..*.****X | 456.278 | | 454.136 | ................................................................X | 454.136 | | 441.197 | ..............................................................****X | 441.197 | | 411.852 | .........................................................*.*****.....*.*****....*.....****X | 521.598 | | 477.693 | ...................................................................X | 477.693 | | 451.093 | ...............................................................***X | 451.093 | | 441.603 | ..............................................................X | 441.603 | | 357.018 | ..................................................****X | 357.018 | | 356.636 | ..................................................*..............*****..X | 473.896 | | 465.648 | .................................................................****X | 465.648 | | 400.392 | ........................................................*****......*....X | 473.650 | | 468.340 | ..................................................................***X | 468.340 | | 440.634 | ..............................................................X | 440.634 | | 426.323 | ............................................................*.****..........*..****X | 520.920 | | 471.023 | ..................................................................X | 471.023 | | 455.946 | ................................................................****X | 455.946 | | 420.843 | ...........................................................X | 420.843 | | 410.620 | .........................................................*****......*.****X | 457.214 | | 448.158 | ...............................................................X | 448.158 | | 438.655 | .............................................................*****.*..****X | 455.312 | | 421.558 | ...........................................................X | 421.558 | | 411.348 | .........................................................*****.X | 415.230 | | 396.086 | .......................................................***X | 396.086 | | 393.296 | .......................................................*....*****....*....****X | 478.772 | | 463.873 | .................................................................X | 463.873 | | 426.568 | ............................................................*****.*..****X | 451.811 | | 428.001 | ............................................................*X | 428.501 | | 419.279 | ...........................................................*****.******......*****...X | 490.287 | | 481.599 | ...................................................................****X | 481.599 | | 470.323 | ..................................................................*..****X | 485.697 | | 478.648 | ...................................................................*****.....*..****X | 527.748 | +---------+---------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 22:55:14,953 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 68.825 | 6437847040 | 4178567168 | 2551963648 / 7511588864 | 125873 / 251078 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 72.1354644089 | 6437847040 | 4143759360 | 2573205504 / 7511588864 | 125608 / 255022 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 57.4492908215 | 6437847040 | 3778850816 | 3050917888 / 9530753024 | 126313 / 247341 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 62.2406293228 | 6437847040 | 4161544192 | 2575179776 / 7511588864 | 126253 / 254748 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 22:55:25,842 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 22:56:35,967 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 158670496 | 249050139 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 157501312 | 275267651 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 438166144 | 443681352 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-17 22:56:35,967 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #18 test_data_load_collections_with_rebalance_in_out ========= 2021-05-17 22:56:44,073 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-17 22:56:44,073 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-17 22:56:47,736 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 22:56:47,956 | test | INFO | pool-20-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 15.3883333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 18.466358894 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 16.8272115192 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.87 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 22:56:47,992 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:56:53,381 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 22:56:56,563 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 22:56:58,479 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.77 2021-05-17 22:57:03,594 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 2.37 2021-05-17 22:57:08,697 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 4.32 2021-05-17 22:57:13,809 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 6.81 2021-05-17 22:57:18,898 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 10.64 2021-05-17 22:57:24,012 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 14.27 2021-05-17 22:57:29,125 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 17.71 2021-05-17 22:57:34,288 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 20.81 2021-05-17 22:57:39,390 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 23.66 2021-05-17 22:57:42,187 | test | INFO | pool-20-thread-22 | [table_view:display:72] Keys failed in bucket1:8G6TW2bLahaBHCBeB-54-86000:rff%baT2EL7Zu-54-268000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:57:42,188 | test | INFO | pool-20-thread-22 | [table_view:display:72] Keys failed in bucket1:8G6TW2bLahaBHCBeB-54-86000:rff%baT2EL7Zu-54-268000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-251 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:57:44,510 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 26.86 2021-05-17 22:57:45,894 | test | INFO | pool-20-thread-2 | [table_view:display:72] Keys failed in bucket1:jk8k5b3c%n3AFXdNEZh-54-495000:tmNpSpe44bIH-54-903000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:57:45,895 | test | INFO | pool-20-thread-2 | [table_view:display:72] Keys failed in bucket1:jk8k5b3c%n3AFXdNEZh-54-495000:tmNpSpe44bIH-54-903000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-523 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:57:49,631 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 30.31 2021-05-17 22:57:54,773 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:57:59,861 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:58:04,980 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 22:58:10,325 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 34.95 2021-05-17 22:58:15,460 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 38.41 2021-05-17 22:58:20,582 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.93 2021-05-17 22:58:25,710 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 44.48 2021-05-17 22:58:30,811 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 47.44 2021-05-17 22:58:35,928 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 51.21 2021-05-17 22:58:41,015 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 54.39 2021-05-17 22:58:46,131 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 57.5 2021-05-17 22:58:51,240 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 60.86 2021-05-17 22:58:55,387 | test | INFO | pool-20-thread-23 | [table_view:display:72] Keys failed in bucket2:2F2PaKbe4_qg-51-757000:PZ6xZlR-51-985000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,387 | test | INFO | pool-20-thread-16 | [table_view:display:72] Keys failed in bucket2:IzTkjj9kBszw-DKr-53-322000:Glk-53-457000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,388 | test | INFO | pool-20-thread-20 | [table_view:display:72] Keys failed in bucket2:HUO6bu9q4gbdyTFutc7d-52-551000:fel9V_-G-d-52-925000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,388 | test | INFO | pool-20-thread-23 | [table_view:display:72] Keys failed in bucket2:2F2PaKbe4_qg-51-757000:PZ6xZlR-51-985000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,388 | test | INFO | pool-20-thread-16 | [table_view:display:72] Keys failed in bucket2:IzTkjj9kBszw-DKr-53-322000:Glk-53-457000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,388 | test | INFO | pool-20-thread-20 | [table_view:display:72] Keys failed in bucket2:HUO6bu9q4gbdyTFutc7d-52-551000:fel9V_-G-d-52-925000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,388 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in bucket2:IzTkjj9kBszw-DKr-53-322000:uT5nJpNrkteN5-8CU-53-458000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,390 | test | INFO | pool-20-thread-7 | [table_view:display:72] Keys failed in bucket2:2F2PaKbe4_qg-51-757000:0WUH4-51-985000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,391 | test | INFO | pool-20-thread-7 | [table_view:display:72] Keys failed in bucket2:2F2PaKbe4_qg-51-757000:0WUH4-51-985000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:55,392 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in bucket2:IzTkjj9kBszw-DKr-53-322000:uT5nJpNrkteN5-8CU-53-458000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2654 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:58:56,316 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 63.39 2021-05-17 22:59:01,404 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.89 2021-05-17 22:59:06,548 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 22:59:11,625 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 22:59:16,700 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 67.33 2021-05-17 22:59:21,799 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 69.09 2021-05-17 22:59:26,904 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 71.08 2021-05-17 22:59:31,992 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 73.14 2021-05-17 22:59:37,075 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 75.03 2021-05-17 22:59:42,190 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 76.64 2021-05-17 22:59:43,884 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:SljOu4LWZiQpBQu%u1-25-382000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-689 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:59:45,125 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 22:59:47,299 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 78.84 2021-05-17 22:59:48,035 | test | INFO | pool-20-thread-26 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:ennS8eY5y9W-25-381000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-858 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 22:59:52,440 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 81.41 2021-05-17 22:59:57,536 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 83.24 2021-05-17 22:59:57,687 | test | INFO | pool-20-thread-25 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:SljOu4LWZiQpBQu%u1-25-382000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-892 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:00,648 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:ZZXSKjZ57z_-25-383000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-557 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:02,638 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 84.75 2021-05-17 23:00:07,726 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 86.76 2021-05-17 23:00:11,243 | test | INFO | pool-20-thread-11 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:mOERYSMG-25-382000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:12,295 | test | INFO | pool-20-thread-23 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:N0fY%UYB-eGhKS-25-383000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-122 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-6 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:12,309 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:00:12,311 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:00:12,315 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:00:12,826 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 88.75 2021-05-17 23:00:13,733 | test | INFO | pool-20-thread-3 | [table_view:display:72] Keys failed in default:72NaouVDeJsI%-55-890000:3L5k-56-110000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:13,734 | test | INFO | pool-20-thread-3 | [table_view:display:72] Keys failed in default:72NaouVDeJsI%-55-890000:3L5k-56-110000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1901 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1891 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:17,950 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 90.48 2021-05-17 23:00:20,793 | test | INFO | pool-20-thread-28 | [table_view:display:72] Keys failed in default:zNYr8DfOzfk-25-380000:N0fY%UYB-eGhKS-25-383000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-759 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:23,042 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 92.02 2021-05-17 23:00:28,130 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 94.1 2021-05-17 23:00:33,217 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 96.51 2021-05-17 23:00:38,302 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 99.22 2021-05-17 23:00:43,381 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:00:48,457 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:00:49,549 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in default:Rz-a_LtQmQeUy_PBFlb-25-384000:pusJAP-25-386000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2574 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2531 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:49,551 | test | INFO | pool-20-thread-12 | [table_view:display:72] Keys failed in default:Rz-a_LtQmQeUy_PBFlb-25-384000:pusJAP-25-386000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2574 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2531 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:52,134 | test | INFO | pool-20-thread-27 | [table_view:display:72] Keys failed in default:YzMB8cRg13G-55-86000:c2feFkFN-55-231000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2567 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:52,137 | test | INFO | pool-20-thread-27 | [table_view:display:72] Keys failed in default:YzMB8cRg13G-55-86000:c2feFkFN-55-231000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2567 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:00:58,565 | test | INFO | pool-20-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:00:58,676 | test | INFO | pool-20-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 250.721000195 sec 2021-05-17 23:03:56,730 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:03:56,733 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:03:56,734 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:03:56,734 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-17 23:04:04,812 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-17 23:04:04,828 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 23:04:04,829 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-17 23:04:29,141 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 23:04:29,141 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 23:04:29,144 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-17 23:04:29,144 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 2 docs from 0 to 500 2021-05-17 23:04:29,151 | test | ERROR | pool-20-thread-19 | [task:call:3809] Failed to load 2 docs from 2500 to 2900 2021-05-17 23:10:14,778 | test | INFO | pool-20-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.*.....***X | 10.092 | | 10.072 | .....****...........*....***.....*.........***.....*....****...X | 77.301 | | 70.514 | ..........................................***X | 70.514 | | 66.033 | .......................................*.................*.........***..X | 110.426 | | 75.523 | .............................................****...*...........................***X | 123.000 | | 117.869 | .......................................................................X | 117.869 | | 114.560 | .....................................................................***...****...*.****.*......................****X | 161.737 | | 132.114 | ................................................................................X | 132.114 | | 116.302 | ......................................................................**X | 116.302 | | 106.498 | ................................................................X | 106.498 | | 90.296 | ......................................................***X | 90.296 | | 80.212 | ................................................*..***......X | 93.057 | | 90.953 | .......................................................****.........*.......***.................*...***X | 149.860 | | 128.047 | ..............................................................................X | 128.047 | | 119.027 | ........................................................................***X | 119.027 | | 110.924 | ...................................................................X | 110.924 | | 75.790 | .............................................***X | 75.790 | | 67.406 | ........................................X | 67.406 | | 63.303 | ......................................****....*...........*****........*...*****.........****X | 120.774 | | 113.122 | ....................................................................X | 113.122 | | 104.322 | ...............................................................X | 104.322 | | 102.114 | ..............................................................****.*..X | 107.587 | | 107.506 | .................................................................****..*...*.***X | 117.600 | | 111.802 | ....................................................................X | 111.802 | | 100.402 | .............................................................X | 100.402 | | 94.002 | .........................................................***X | 94.002 | | 75.786 | .............................................X | 75.786 | | 72.389 | ...........................................*****......*..****X | 85.212 | | 66.800 | ........................................X | 66.800 | | 57.502 | ..................................****X | 57.502 | | 55.593 | .................................******.*****X | 57.512 | | 33.199 | ...................X | 33.199 | | 4.799 | .***X | 4.799 | | 0.800 | X | 0.800 | | 0.000 | *******************.****X | 2.200 | | 1.500 | X | 1.500 | | 1.000 | **********.*****X | 3.000 | | 1.600 | X | 1.600 | | 1.500 | ****X | 1.500 | | 0.700 | X | 0.700 | | 0.000 | **********X | 1.500 | | 1.500 | X | 1.500 | | 1.500 | *****.****X | 1.699 | | 1.400 | X | 1.400 | | 1.200 | *****X | 1.501 | | 1.400 | *****.*****X | 1.952 | | 1.753 | X | 1.753 | | 1.701 | ****X | 1.901 | | 1.800 | *****X | 1.800 | | 1.338 | *****X | 1.338 | | 1.101 | ******.*****X | 2.003 | | 1.800 | ***********X | 2.699 | | 2.400 | ****X | 2.400 | | 1.801 | *****X | 1.801 | | 0.999 | ******..*.****X | 5.497 | | 3.093 | X | 3.093 | | 0.500 | *****.*****X | 3.201 | | 1.800 | **********X | 2.599 | | 1.698 | X | 1.698 | | 1.100 | **********X | 1.301 | | 1.300 | *.****X | 1.699 | | 1.599 | X | 1.599 | | 1.400 | ****X | 1.400 | | 0.999 | X | 0.999 | | 0.899 | *****.****X | 2.298 | | 2.098 | X | 2.098 | | 0.999 | ******.****X | 1.899 | | 1.898 | X | 1.898 | | 1.299 | *****.*.*****X | 3.898 | | 3.698 | .****X | 3.698 | | 1.499 | ******.*********X | 2.099 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:10:15,796 | test | INFO | pool-20-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****.***X | 3.400 | | 2.597 | *........***..........*...***........*....................***X | 129.782 | | 129.034 | ................................................X | 129.034 | | 113.162 | ..........................................***...*........*...........***X | 169.721 | | 162.579 | .............................................................*.***X | 165.182 | | 143.568 | ......................................................****X | 143.568 | | 128.925 | ................................................****..............****...X | 172.598 | | 161.378 | ............................................................****...X | 167.936 | | 162.010 | .............................................................****............*..............**X | 230.244 | | 220.068 | ...................................................................................**................X | 261.019 | | 252.805 | ...............................................................................................**X | 252.805 | | 217.096 | ..................................................................................X | 217.096 | | 187.791 | ......................................................................****.............*..***X | 225.135 | | 214.368 | .................................................................................***X | 214.368 | | 51.484 | ..................X | 51.484 | | 1.499 | ***X | 1.499 | | 0.800 | X | 0.800 | | 0.700 | ********X | 2.201 | | 0.800 | ****.****X | 5.200 | | 4.199 | X | 4.199 | | 2.600 | ****X | 2.600 | | 0.200 | X | 0.200 | | 0.000 | ***********.*****X | 2.701 | | 0.100 | ******.****X | 3.000 | | 2.300 | X | 2.300 | | 1.000 | ******.*****X | 3.499 | | 0.500 | *****...*................*****..X | 57.198 | | 56.094 | ....................X | 56.094 | | 52.394 | ...................***X | 52.394 | | 51.695 | ..................*..*..*****..*****...*...****X | 83.205 | | 77.893 | ............................*..****X | 81.094 | | 74.500 | ...........................X | 74.500 | | 74.498 | ...........................****X | 74.498 | | 60.803 | ......................X | 60.803 | | 39.003 | .............****X | 39.003 | | 17.002 | .....X | 17.002 | | 2.599 | ****X | 2.599 | | 1.300 | X | 1.300 | | 0.700 | ***************X | 2.300 | | 1.600 | X | 1.600 | | 1.100 | *****.*.****X | 5.501 | | 3.901 | X | 3.901 | | 2.000 | ***X | 2.000 | | 1.600 | X | 1.600 | | 1.599 | **********X | 2.302 | | 1.902 | X | 1.902 | | 1.100 | *****.**.***X | 6.284 | | 4.583 | X | 4.583 | | 2.583 | X | 2.583 | | 1.000 | ***X | 1.000 | | 0.900 | *****X | 1.300 | | 1.200 | *******.***X | 3.803 | | 3.100 | X | 3.100 | | 2.399 | ****X | 2.399 | | 2.098 | X | 2.098 | | 1.801 | ****X | 1.801 | | 1.801 | X | 1.801 | | 1.601 | ******.***********X | 4.701 | | 3.000 | *****X | 3.100 | | 3.100 | ****X | 3.100 | | 2.197 | X | 2.197 | | 1.198 | X | 1.198 | | 0.699 | *****.*****X | 4.801 | | 3.900 | ****X | 3.900 | | 2.398 | X | 2.398 | | 1.700 | ****X | 1.700 | | 1.600 | *******.****X | 2.798 | | 1.598 | X | 1.598 | | 0.700 | **********X | 1.400 | | 0.399 | X | 0.399 | | 0.100 | ******.***X | 3.597 | | 3.196 | X | 3.196 | | 1.299 | ******.****X | 4.098 | | 3.297 | X | 3.297 | | 2.199 | *****.******.*****X | 7.097 | | 1.700 | *****.*****X | 4.397 | | 4.197 | X | 4.197 | | 3.598 | ****X | 3.598 | +---------+--------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:10:16,436 | test | INFO | pool-20-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....*...........***X | 119.379 | | 31.543 | ...****...................................*...................................**X | 565.567 | | 463.495 | ............................................................X | 463.495 | | 436.807 | ........................................................**X | 436.807 | | 375.469 | ................................................X | 375.469 | | 300.526 | ......................................****............................................*................***X | 759.432 | | 648.317 | ....................................................................................X | 648.317 | | 382.655 | .................................................***X | 382.655 | | 282.875 | ....................................***X | 282.875 | | 259.713 | .................................X | 259.713 | | 197.756 | .........................****.......................***X | 374.110 | | 300.362 | ......................................X | 300.362 | | 208.584 | ..........................**X | 208.584 | | 179.810 | ......................X | 179.810 | | 148.774 | ..................****.*.*****.................................***X | 414.248 | | 376.335 | ................................................X | 376.335 | | 265.077 | .................................***X | 265.077 | | 251.913 | ................................X | 251.913 | | 175.671 | ......................***X | 175.671 | | 161.440 | ....................X | 161.440 | | 138.057 | .................X | 138.057 | | 122.978 | ...............***..*....****.........................................****X | 479.715 | | 350.050 | .............................................X | 350.050 | | 318.267 | ........................................***X | 318.267 | | 288.318 | ....................................****X | 288.318 | | 228.796 | .............................*****...........*......*X | 363.269 | | 357.777 | ..............................................**..*..*....****..*...........*****.X | 524.108 | | 433.772 | ........................................................****X | 433.772 | | 338.870 | ...........................................X | 338.870 | | 268.015 | ..................................*****....*....*****.........*.....X | 437.311 | | 437.311 | ........................................................****......*.****X | 492.128 | | 468.697 | ............................................................X | 468.697 | | 452.062 | ..........................................................****X | 452.062 | | 247.193 | ...............................X | 247.193 | | 204.704 | .........................*****...........X | 288.166 | | 286.266 | ....................................***X | 286.266 | | 277.974 | ...................................*......****X | 319.969 | | 293.448 | .....................................X | 293.448 | | 183.886 | .......................*****...*.....*****.*..X | 273.278 | | 273.278 | ..................................*X | 273.278 | | 273.278 | ..................................**.*...*****.....................*.....*****.X | 506.362 | | 489.449 | ...............................................................*****.....*.*X | 537.237 | | 537.237 | .....................................................................*X | 537.237 | | 537.237 | .....................................................................X | 537.237 | | 460.014 | ...........................................................X | 460.014 | | 428.557 | .......................................................****....*..****X | 471.611 | | 465.349 | ............................................................*...*****X | 492.155 | | 486.197 | ...............................................................****X | 486.197 | | 486.069 | ...............................................................******......****X | 538.526 | | 483.191 | ..............................................................*.X | 487.442 | | 457.818 | ...........................................................****.*..****X | 483.376 | | 442.820 | .........................................................X | 442.820 | | 435.479 | ........................................................*****....X | 470.305 | | 469.941 | ............................................................***X | 469.941 | | 446.628 | .........................................................X | 446.628 | | 407.712 | ....................................................*****..*......****X | 466.217 | | 443.617 | .........................................................X | 443.617 | | 390.383 | ..................................................****.*..................*****X | 534.861 | | 397.887 | ...................................................******.****X | 404.714 | | 350.270 | .............................................X | 350.270 | | 329.260 | ..........................................*****..........*......****X | 450.846 | | 394.758 | ..................................................X | 394.758 | | 338.777 | ...........................................*****..........*....****X | 440.538 | | 438.360 | ........................................................*.***X | 446.308 | | 442.750 | .........................................................*.*.***X | 457.935 | | 453.834 | ..........................................................X | 453.834 | | 444.499 | .........................................................****X | 444.499 | | 430.571 | .......................................................X | 430.571 | | 399.244 | ...................................................*****....*.....*****.*..****X | 490.982 | | 487.877 | ...............................................................X | 487.877 | | 486.817 | ...............................................................X | 486.817 | | 465.487 | ............................................................****.......*....*X | 550.501 | | 550.501 | .......................................................................**X | 550.501 | | 488.742 | ...............................................................X | 488.742 | | 462.142 | ...........................................................****X | 462.142 | | 431.005 | .......................................................X | 431.005 | | 424.199 | ......................................................*****......*..****X | 483.988 | | 405.965 | ....................................................X | 405.965 | | 389.201 | ..................................................*****............X | 481.380 | | 472.598 | .............................................................*****......****X | 521.828 | | 516.519 | ...................................................................X | 516.519 | | 478.609 | ..............................................................*****..*.........*****..X | 579.360 | | 486.603 | ...............................................................*****...............*........****X | 667.820 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:10:16,500 | test | WARNING | Thread-45718 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:jk8k5b3c%n3AFXdNEZh-54-495000:tmNpSpe44bIH-54-903000 create failed keys from task: [u'test_collections-523'] 2021-05-17 23:10:16,505 | test | WARNING | Thread-45716 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8G6TW2bLahaBHCBeB-54-86000:rff%baT2EL7Zu-54-268000 create failed keys from task: [u'test_collections-251'] 2021-05-17 23:10:16,529 | test | WARNING | Thread-45833 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:2F2PaKbe4_qg-51-757000:PZ6xZlR-51-985000 create failed keys from task: [u'test_collections-2654'] 2021-05-17 23:10:16,582 | test | WARNING | Thread-46022 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:72NaouVDeJsI%-55-890000:3L5k-56-110000 create failed keys from task: [u'test_collections-1901', u'test_collections-1891'] 2021-05-17 23:10:16,592 | test | WARNING | Thread-46060 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Rz-a_LtQmQeUy_PBFlb-25-384000:pusJAP-25-386000 create failed keys from task: [u'test_collections-2574', u'test_collections-2531'] 2021-05-17 23:10:16,595 | test | WARNING | Thread-46066 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:N0fY%UYB-eGhKS-25-383000 update failed keys from task: [u'test_collections-759'] 2021-05-17 23:10:16,595 | test | WARNING | Thread-46065 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:SljOu4LWZiQpBQu%u1-25-382000 update failed keys from task: [u'test_collections-689', u'test_collections-892'] 2021-05-17 23:10:16,605 | test | WARNING | Thread-46084 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:mOERYSMG-25-382000 update failed keys from task: [u'test_collections-829'] 2021-05-17 23:10:16,630 | test | WARNING | Thread-46066 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:N0fY%UYB-eGhKS-25-383000 delete failed keys from task: [u'test_collections-122', u'test_collections-6'] 2021-05-17 23:10:16,638 | test | WARNING | Thread-46093 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:ennS8eY5y9W-25-381000 update failed keys from task: [u'test_collections-858'] 2021-05-17 23:10:16,638 | test | WARNING | Thread-46085 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:zNYr8DfOzfk-25-380000:ZZXSKjZ57z_-25-383000 update failed keys from task: [u'test_collections-557'] 2021-05-17 23:10:16,674 | test | WARNING | Thread-45834 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:2F2PaKbe4_qg-51-757000:0WUH4-51-985000 create failed keys from task: [u'test_collections-2654'] 2021-05-17 23:10:16,674 | test | WARNING | Thread-46136 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:YzMB8cRg13G-55-86000:c2feFkFN-55-231000 create failed keys from task: [u'test_collections-2567'] 2021-05-17 23:10:16,707 | test | WARNING | Thread-45909 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:IzTkjj9kBszw-DKr-53-322000:uT5nJpNrkteN5-8CU-53-458000 create failed keys from task: [u'test_collections-2654'] 2021-05-17 23:10:16,717 | test | WARNING | Thread-45910 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:IzTkjj9kBszw-DKr-53-322000:Glk-53-457000 create failed keys from task: [u'test_collections-2654'] 2021-05-17 23:10:16,717 | test | WARNING | Thread-45912 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:HUO6bu9q4gbdyTFutc7d-52-551000:fel9V_-G-d-52-925000 create failed keys from task: [u'test_collections-2654'] 2021-05-17 23:10:27,671 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 23:11:32,907 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 23:11:32,907 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 23:11:58,101 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 23:11:58,101 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 23:12:18,783 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 23:12:18,785 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 23:13:17,278 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1320002079 seconds to complete 2021-05-17 23:13:17,292 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2482.912s OK During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 18 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_18 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_19 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_rebalance_cycles,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE' Test Input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_19', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 19, 'bulk_api_crud': 'True'} test_rebalance_cycles (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 23:13:27,157 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #19 test_rebalance_cycles ========= 2021-05-17 23:13:27,838 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #19 test_rebalance_cycles ========= 2021-05-17 23:13:28,309 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5cf6e3 type=client open_count=1 channel=[id: 0x18f00d9f, 0.0.0.0/0.0.0.0:39843] timeout=300.0>) 2021-05-17 23:13:50,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5cf6e4 type=client open_count=1 channel=[id: 0x5f94fb7e, 0.0.0.0/0.0.0.0:49060] timeout=300.0>) 2021-05-17 23:13:52,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5cf6e5 type=client open_count=1 channel=[id: 0xf81c5c6b, 0.0.0.0/0.0.0.0:59033] timeout=300.0>) 2021-05-17 23:13:54,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5cf6e6 type=client open_count=1 channel=[id: 0x25b115e2, 0.0.0.0/0.0.0.0:41623] timeout=300.0>) 2021-05-17 23:14:17,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5cf6e7 type=client open_count=1 channel=[id: 0xe56432bb, 0.0.0.0/0.0.0.0:53004] timeout=300.0>) 2021-05-17 23:14:19,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5cf6e8 type=client open_count=1 channel=[id: 0x3597d45a, 0.0.0.0/0.0.0.0:50219] timeout=300.0>) 2021-05-17 23:14:21,433 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5cf6e9 type=client open_count=1 channel=[id: 0xed62db5c, 0.0.0.0/0.0.0.0:37710] timeout=300.0>) 2021-05-17 23:14:45,161 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5cf6ea type=client open_count=1 channel=[id: 0x4be0b573, 0.0.0.0/0.0.0.0:48005] timeout=300.0>) 2021-05-17 23:14:47,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5cf6eb type=client open_count=1 channel=[id: 0xb751908b, 0.0.0.0/0.0.0.0:41324] timeout=300.0>) 2021-05-17 23:14:49,197 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5cf6ec type=client open_count=1 channel=[id: 0xae89abf3, 0.0.0.0/0.0.0.0:59616] timeout=300.0>) 2021-05-17 23:15:13,479 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5cf6ed type=client open_count=1 channel=[id: 0x9c396893, 0.0.0.0/0.0.0.0:35661] timeout=300.0>) 2021-05-17 23:15:15,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5cf6ee type=client open_count=1 channel=[id: 0x9356b835, 0.0.0.0/0.0.0.0:54039] timeout=300.0>) 2021-05-17 23:15:17,515 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5cf6ef type=client open_count=1 channel=[id: 0x7d1d18cc, 0.0.0.0/0.0.0.0:52975] timeout=300.0>) 2021-05-17 23:15:41,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5cf6f0 type=client open_count=1 channel=[id: 0x24844648, 0.0.0.0/0.0.0.0:53156] timeout=300.0>) 2021-05-17 23:15:43,346 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5cf6f1 type=client open_count=1 channel=[id: 0x2150a9cb, 0.0.0.0/0.0.0.0:36087] timeout=300.0>) 2021-05-17 23:15:45,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 23:16:03,289 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 23:16:17,049 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 23:16:17,072 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #19 test_rebalance_cycles ========= 2021-05-17 23:16:17,115 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #19 test_rebalance_cycles ========= 2021-05-17 23:16:27,555 | test | INFO | pool-21-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.37659372344 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:16:27,599 | test | INFO | pool-21-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:16:37,673 | test | INFO | pool-21-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:16:37,700 | test | INFO | pool-21-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1460001469 sec 2021-05-17 23:16:37,966 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 23:16:38,181 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 23:16:38,325 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | 94DUwDoaU-38-10000 | couchbase | 2 | none | 0 | 0 | 19167969280 | 0 | 0 | +--------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 23:16:38,325 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 23:16:41,668 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:16:41,674 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:17:27,309 | test | INFO | pool-21-thread-9 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ | 223.911 | .............................................................................................*******......******X | 237.494 | | 221.105 | ............................................................................................******......X | 235.928 | +---------+-------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:17:27,374 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 59.2709878498 | 6437847040 | 4603944960 | 2089259008 / 7511588864 | 2394 / 4827 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 60.7316666667 | 6437847040 | 4652109824 | 2031337472 / 7511588864 | 2373 / 4751 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 60.365 | 6437847040 | 4212203520 | 2561421312 / 9530753024 | 2390 / 4679 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 52.0865971134 | 6437847040 | 4642050048 | 2064863232 / 7511588864 | 2356 / 4753 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 23:17:31,578 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 23:17:48,209 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | 94DUwDoaU-38-10000 | couchbase | 2 | none | 0 | 10000 | 19167969280 | 116697808 | 180845306 | +--------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 23:17:48,210 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #19 test_rebalance_cycles ========= 2021-05-17 23:17:53,253 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 0 2021-05-17 23:17:53,328 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:17:53,334 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:17:53,371 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:17:55,407 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:17:57,431 | test | INFO | pool-21-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 10.7786926231 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 13.98620046 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 12.7602126702 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 11.5883333333 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:17:57,453 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:17:57,525 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:18:02,562 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.47 2021-05-17 23:18:03,815 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-700 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:05,788 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-849 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-803 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-846 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-932 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-993 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:05,851 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2750 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-2730 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:07,660 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 14.05 2021-05-17 23:18:08,319 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1039 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1018 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1188 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:08,878 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2831 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2963 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-2878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-2837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:12,733 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 23.76 2021-05-17 23:18:13,056 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1348 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1271 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1222 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:14,604 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3116 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3039 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3142 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3031 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3075 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3086 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3166 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3154 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3123 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:15,397 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1459 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:17,450 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3259 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:17,823 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 35.25 2021-05-17 23:18:19,640 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3520 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3478 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:19,713 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1665 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:21,895 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3776 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-3767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:22,505 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1915 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1928 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1984 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-1885 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1820 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-1957 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-1870 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:22,903 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 49.58 2021-05-17 23:18:24,049 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3823 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:24,051 | test | ERROR | pool-21-thread-18 | [task:call:3809] Failed to load 21 docs from 2000 to 3920 2021-05-17 23:18:24,052 | test | ERROR | pool-21-thread-18 | [task:call:3809] Failed to load 27 docs from 0 to 2000 2021-05-17 23:18:25,539 | test | INFO | pool-21-thread-19 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*****X | 252.706 | | 175.200 | ....................................................................****X | 175.200 | | 163.104 | ...............................................................****X | 163.104 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:18:27,984 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.7 2021-05-17 23:18:33,071 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: running, progress: 95.41 2021-05-17 23:18:43,167 | test | INFO | pool-21-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:18:43,200 | test | INFO | pool-21-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 45.7680001259 sec 2021-05-17 23:18:43,246 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:18:43,253 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:18:43,286 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:18:43,361 | test | INFO | pool-21-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 24.2725242492 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 49.9233358888 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 15.9378020733 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 71.458093762 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.61467951068 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:18:43,394 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:18:45,427 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:18:48,476 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 3.93 2021-05-17 23:18:53,513 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2446 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2402 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2568 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2527 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2504 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2592 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2494 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2562 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:53,592 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 17.79 2021-05-17 23:18:54,219 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4392 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4337 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4506 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4331 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4496 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4445 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4443 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4389 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:55,813 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2633 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-2669 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:56,154 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12390 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-12200 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:56,154 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12390 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12200 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:18:58,673 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 28.13 2021-05-17 23:19:00,538 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2828 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2805 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2819 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2909 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2995 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2874 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2899 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-2989 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:02,405 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:02,407 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12568 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:03,765 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 37.61 2021-05-17 23:19:03,921 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4779 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4832 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4739 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-4796 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:05,081 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3073 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3007 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3021 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3197 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3020 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3047 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3069 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:07,444 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3314 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3284 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:07,920 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4920 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5024 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:08,309 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:08,309 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12979 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:08,851 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 47.08 2021-05-17 23:19:09,605 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3519 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3561 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3564 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3489 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:10,150 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5199 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:11,555 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3707 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-3704 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3694 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-3697 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:12,071 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5350 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5472 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5422 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5467 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:13,743 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-5656 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:13,950 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 55.35 2021-05-17 23:19:15,357 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5751 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:19:15,358 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 24 docs from 3920 to 5763 2021-05-17 23:19:15,358 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 35 docs from 2000 to 3920 2021-05-17 23:19:15,358 | test | ERROR | pool-21-thread-25 | [task:call:3809] Failed to load 4 docs from 11600 to 13136 2021-05-17 23:19:16,496 | test | INFO | pool-21-thread-26 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................****X | 257.500 | | 145.888 | .......................................................****X | 145.888 | | 127.004 | ................................................****X | 127.004 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:19:19,029 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 68.91 2021-05-17 23:19:24,115 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 84.12 2021-05-17 23:19:29,190 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 98.09 2021-05-17 23:19:34,265 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:19:39,325 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:19:49,407 | test | INFO | pool-21-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:19:49,552 | test | INFO | pool-21-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 66.1900000572 sec 2021-05-17 23:19:49,552 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 1 2021-05-17 23:19:49,599 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:19:49,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:19:49,640 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:19:51,674 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:19:53,707 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:19:53,806 | test | INFO | pool-21-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 6.17312756241 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 5.80695701452 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 7.05666666667 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:19:53,826 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:19:55,821 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:19:58,895 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.9 2021-05-17 23:20:03,970 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 13.16 2021-05-17 23:20:05,316 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6433 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6476 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6366 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6558 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6436 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6414 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:05,589 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4658 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4713 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4702 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4549 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4538 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4714 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4618 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4629 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4718 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4683 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4595 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4684 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4692 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4688 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4586 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4597 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:06,496 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:06,497 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13920 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:09,048 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 23.2 2021-05-17 23:20:13,503 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4823 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4900 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4866 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4837 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4890 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4742 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:14,151 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 33.48 2021-05-17 23:20:17,486 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6947 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6857 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6807 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6796 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6884 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6855 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6877 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6842 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-6951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:19,230 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 47.19 2021-05-17 23:20:19,494 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5098 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5108 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4927 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5066 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5023 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-4962 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:21,341 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5141 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:22,709 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7030 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7155 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-7003 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:23,959 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:24,305 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 59.33 2021-05-17 23:20:27,233 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7270 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7231 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7164 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7217 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:27,493 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5574 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5553 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-5567 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:28,306 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:28,308 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14594 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:29,190 | test | ERROR | pool-21-thread-5 | [task:call:3809] Failed to load 27 docs from 5763 to 7532 2021-05-17 23:20:29,341 | test | ERROR | pool-21-thread-5 | [task:call:3809] Failed to load 35 docs from 3920 to 5763 2021-05-17 23:20:29,341 | test | ERROR | pool-21-thread-5 | [task:call:3809] Failed to load 2 docs from 13136 to 14610 2021-05-17 23:20:29,394 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 73.12 2021-05-17 23:20:29,829 | test | INFO | pool-21-thread-2 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****..................................................................*..................................****X | 304.600 | | 103.984 | .................................****X | 103.984 | | 91.399 | .............................*****.............*....X | 144.332 | +---------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:20:34,470 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: running, progress: 91.05 2021-05-17 23:20:44,562 | test | INFO | pool-21-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:20:44,595 | test | INFO | pool-21-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 50.7899999619 sec 2021-05-17 23:20:44,642 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:20:44,648 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:20:44,684 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:20:44,779 | test | INFO | pool-21-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 41.3543107615 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 70.2588290195 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 24.5579240346 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 78.3836936051 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:20:44,823 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:20:47,069 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:20:49,963 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 5.39 2021-05-17 23:20:50,515 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6050 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:55,055 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.26 2021-05-17 23:20:56,345 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8096 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8050 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8046 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7991 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8131 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8093 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8028 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7960 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-8015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8103 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7987 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8106 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-7952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:20:56,891 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6186 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6284 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6329 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6199 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6314 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6336 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6204 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:00,144 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 28.9 2021-05-17 23:21:00,657 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6388 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6465 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6426 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6558 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:00,717 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15272 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:00,719 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15272 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:05,243 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.49 2021-05-17 23:21:06,727 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8382 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:07,529 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6615 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6609 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6700 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:10,331 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 54.48 2021-05-17 23:21:15,062 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8793 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-8755 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:15,395 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.54 2021-05-17 23:21:16,687 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7078 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-7003 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7046 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-6998 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:17,852 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9009 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:18,246 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15855 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:18,247 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15855 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:20,308 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7186 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7258 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7244 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-7229 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:20,471 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 76.55 2021-05-17 23:21:21,128 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9142 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:21,130 | test | ERROR | pool-21-thread-9 | [task:call:3809] Failed to load 21 docs from 7532 to 9230 2021-05-17 23:21:21,934 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-7475 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:21:21,934 | test | ERROR | pool-21-thread-9 | [task:call:3809] Failed to load 26 docs from 5763 to 7532 2021-05-17 23:21:21,936 | test | ERROR | pool-21-thread-9 | [task:call:3809] Failed to load 2 docs from 14610 to 16025 2021-05-17 23:21:22,112 | test | INFO | pool-21-thread-8 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 39.802 | ....................*..................*.............................................................****X | 184.681 | | 166.503 | .........................................................................................X | 166.503 | | 155.117 | ..................................................................................***X | 155.117 | | 123.107 | .................................................................X | 123.107 | | 104.488 | .......................................................****.......*...........*X | 137.179 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:21:25,563 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 92.85 2021-05-17 23:21:30,631 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:21:35,694 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:21:45,792 | test | INFO | pool-21-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:21:45,959 | test | INFO | pool-21-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 61.1800000668 sec 2021-05-17 23:21:45,959 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 2 2021-05-17 23:21:46,016 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:21:46,022 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:21:46,055 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:21:48,085 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:21:50,124 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:21:50,328 | test | INFO | pool-21-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 42.8633333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 5.88343139052 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 6.065 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:21:50,362 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:21:52,250 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:21:55,440 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.9 2021-05-17 23:22:00,536 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 11.79 2021-05-17 23:22:02,707 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9870 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9982 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10015 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9907 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9908 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10024 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10021 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9987 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9857 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9869 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9830 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9897 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-9942 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:02,868 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8174 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8282 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8260 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8193 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8233 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8134 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8276 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8265 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8153 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8312 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8329 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:05,637 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 21.68 2021-05-17 23:22:09,494 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8451 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8491 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8439 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8501 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8369 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:10,710 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.39 2021-05-17 23:22:11,413 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10207 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10214 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10145 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10046 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10055 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10186 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10064 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10062 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10193 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:14,928 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8683 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8640 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8591 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8581 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8700 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8645 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8678 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8569 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-8713 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8597 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:15,799 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 42.53 2021-05-17 23:22:17,026 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10418 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10335 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10324 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10412 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10360 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-10381 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:17,793 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-8826 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:19,397 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10471 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:20,384 | test | INFO | pool-21-thread-21 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9022 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-9023 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9071 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-8975 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:20,825 | test | INFO | pool-21-thread-22 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10778 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:20,882 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 51.71 2021-05-17 23:22:22,009 | test | ERROR | pool-21-thread-15 | [task:call:3809] Failed to load 32 docs from 9230 to 10860 2021-05-17 23:22:22,010 | test | ERROR | pool-21-thread-15 | [task:call:3809] Failed to load 33 docs from 7532 to 9230 2021-05-17 23:22:22,174 | test | INFO | pool-21-thread-20 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..................................................................*..................................***X | 355.564 | | 170.025 | ..............................................X | 170.025 | | 79.134 | .....................*****...**X | 89.769 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:22:25,967 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 67.14 2021-05-17 23:22:31,049 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: running, progress: 83.78 2021-05-17 23:22:41,141 | test | INFO | pool-21-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:22:41,173 | test | INFO | pool-21-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 50.8460001945 sec 2021-05-17 23:22:41,226 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:22:41,232 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:22:41,267 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:22:41,348 | test | INFO | pool-21-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 72.4724157472 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 66.2516874156 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 59.065 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 67.1616914179 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:22:41,384 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:22:41,446 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12200 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:41,480 | test | INFO | pool-21-thread-30 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12390 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:41,483 | test | ERROR | pool-21-thread-17 | [task:call:3809] Failed to load 2 docs from 10860 to 12425 2021-05-17 23:22:43,664 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:22:45,946 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11225 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11137 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:46,470 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 5.39 2021-05-17 23:22:46,621 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9498 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9508 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:50,762 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11260 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11379 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:51,582 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 12.85 2021-05-17 23:22:52,191 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9672 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9680 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9789 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9643 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9710 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9799 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:22:56,713 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 23.75 2021-05-17 23:23:01,039 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11560 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11563 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11525 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11526 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:01,752 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-9894 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9982 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10019 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10006 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10016 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9904 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-9987 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:01,857 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 37.3 2021-05-17 23:23:05,301 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10046 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:05,851 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11774 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-11854 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-11759 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:06,960 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 52.85 2021-05-17 23:23:10,641 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10598 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10473 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:12,038 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 64.37 2021-05-17 23:23:13,043 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12141 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12097 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-12157 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-12139 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-12107 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:15,338 | test | INFO | pool-21-thread-28 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10681 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10661 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-10720 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10711 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-10778 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:16,744 | test | INFO | pool-21-thread-29 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12278 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:16,746 | test | ERROR | pool-21-thread-17 | [task:call:3809] Failed to load 19 docs from 10860 to 12425 2021-05-17 23:23:16,891 | test | ERROR | pool-21-thread-17 | [task:call:3809] Failed to load 24 docs from 9230 to 10860 2021-05-17 23:23:17,108 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 75.74 2021-05-17 23:23:17,203 | test | INFO | pool-21-thread-27 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................****X | 261.380 | | 224.590 | ....................................................................................X | 224.590 | | 131.110 | .................................................***X | 131.110 | | 88.805 | ................................X | 88.805 | | 63.296 | .......................****X | 63.296 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:23:22,256 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 93.08 2021-05-17 23:23:27,367 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:23:32,433 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:23:42,607 | test | INFO | pool-21-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:23:43,767 | test | INFO | pool-21-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 62.4210000038 sec 2021-05-17 23:23:43,769 | test | INFO | MainThread | [collections_rebalance:test_rebalance_cycles:1028] Cycle 3 2021-05-17 23:23:43,828 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:23:43,834 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:23:43,871 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:23:43,894 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:43,921 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:43,957 | test | INFO | pool-21-thread-7 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:23:43,959 | test | ERROR | pool-21-thread-24 | [task:call:3809] Failed to load 3 docs from 12425 to 13928 2021-05-17 23:23:45,904 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:23:47,915 | test | INFO | pool-21-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 5.86156897385 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 10.3620120671 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 6.58844314072 | Cluster node | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:23:47,941 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:23:48,019 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:23:53,045 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.51 2021-05-17 23:23:58,128 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 12.73 2021-05-17 23:23:58,561 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13160 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13052 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13187 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13154 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13176 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13147 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13125 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:00,161 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11563 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11488 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11643 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11632 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11637 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11606 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11518 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:03,216 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 21.48 2021-05-17 23:24:05,993 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13383 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13262 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13396 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13253 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13277 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13232 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13268 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13267 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13245 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13237 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13368 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13349 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13359 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:08,319 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 33.09 2021-05-17 23:24:08,605 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11690 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11671 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11693 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11795 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11785 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11799 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11700 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11689 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11768 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11703 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11829 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11719 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11808 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:10,997 | test | INFO | pool-21-thread-1 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13611 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13478 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-13623 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-13568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:11,473 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-11939 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:13,401 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-17 23:24:13,742 | test | INFO | pool-21-thread-6 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12231 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:14,267 | test | ERROR | pool-21-thread-24 | [task:call:3809] Failed to load 25 docs from 12425 to 13928 2021-05-17 23:24:15,167 | test | ERROR | pool-21-thread-24 | [task:call:3809] Failed to load 22 docs from 10860 to 12425 2021-05-17 23:24:16,101 | test | INFO | pool-21-thread-4 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*****X | 210.300 | | 134.713 | ...............................................................****X | 134.713 | | 113.570 | .....................................................X | 113.570 | | 97.362 | .............................................***X | 97.362 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:24:18,480 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 58.12 2021-05-17 23:24:23,555 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 75.43 2021-05-17 23:24:28,621 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: running, progress: 93.2 2021-05-17 23:24:38,743 | test | INFO | pool-21-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:24:38,796 | test | INFO | pool-21-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 50.8800001144 sec 2021-05-17 23:24:38,854 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:24:38,859 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:24:38,901 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 2 seconds. Reason: wait for rebalance to start 2021-05-17 23:24:38,986 | test | INFO | pool-21-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 66.2505624906 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 44.6618446052 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 38.3076948718 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 53.7086997069 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:24:38,987 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:39,023 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:24:39,032 | test | INFO | pool-21-thread-14 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15272 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:39,049 | test | ERROR | pool-21-thread-3 | [task:call:3809] Failed to load 2 docs from 13928 to 15370 2021-05-17 23:24:41,186 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 10 seconds. Reason: wait before dropping collections using bulk api 2021-05-17 23:24:43,852 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14244 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14224 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:44,148 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 6.29 2021-05-17 23:24:44,904 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-12796 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12730 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12720 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12687 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12776 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12666 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12629 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12717 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:49,249 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 18.36 2021-05-17 23:24:50,773 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14393 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14472 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14471 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14350 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14464 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14467 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14346 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14434 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14478 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14422 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14328 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14409 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:52,709 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13001 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12895 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12983 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12862 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12864 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-12856 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12999 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12938 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12905 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:54,357 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.85 2021-05-17 23:24:59,020 | test | INFO | pool-21-thread-8 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20449 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:59,022 | test | INFO | pool-21-thread-8 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20449 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:24:59,437 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.9 2021-05-17 23:25:01,219 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14684 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14688 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14714 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:02,282 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13070 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13092 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13097 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13063 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13025 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13102 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13049 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13107 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13205 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:03,746 | test | INFO | pool-21-thread-8 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-20711 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +------------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:03,746 | test | INFO | pool-21-thread-8 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-20681 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-20711 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:04,559 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 58.94 2021-05-17 23:25:06,369 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13259 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13237 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13413 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:08,822 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15047 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15014 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15017 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-14943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:09,108 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13456 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-13544 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:09,674 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 69.29 2021-05-17 23:25:13,490 | test | INFO | pool-21-thread-13 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-13650 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13785 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13774 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13817 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:13,529 | test | INFO | pool-21-thread-12 | [table_view:display:72] Keys failed in 94DUwDoaU-38-10000:_default:_default +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-15212 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15257 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-15216 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +------------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 23:25:14,710 | test | ERROR | pool-21-thread-3 | [task:call:3809] Failed to load 25 docs from 13928 to 15370 2021-05-17 23:25:14,803 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 78.9 2021-05-17 23:25:14,878 | test | ERROR | pool-21-thread-3 | [task:call:3809] Failed to load 39 docs from 12425 to 13928 2021-05-17 23:25:14,878 | test | ERROR | pool-21-thread-3 | [task:call:3809] Failed to load 3 docs from 19940 to 21142 2021-05-17 23:25:15,750 | test | INFO | pool-21-thread-11 | [table_view:display:72] Ops trend for bucket '94DUwDoaU-38-10000' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................***X | 264.973 | | 264.973 | ..................................................................................................X | 264.973 | | 84.102 | ..............................*****..X | 87.698 | | 87.598 | ................................***X | 87.598 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:25:19,868 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 94.84 2021-05-17 23:25:24,937 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:25:30,019 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 23:25:40,124 | test | INFO | pool-21-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:25:40,282 | test | INFO | pool-21-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 61.2960000038 sec 2021-05-17 23:25:41,240 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: 94DUwDoaU-38-10000, Active Resident ratio(DGM): 100% 2021-05-17 23:25:41,242 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: 94DUwDoaU-38-10000, Replica Resident ratio(DGM): 100% 2021-05-17 23:26:08,618 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1700000763 seconds to complete 2021-05-17 23:26:08,634 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 771.254s OK During the test, Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 19 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_19 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_20 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_Orchestrator_Node_failover,nodes_init=4,nodes_in=2,durability=MAJORITY_AND_PERSIST_TO_ACTIVE,replicas=2,bucket_spec=single_bucket.default,num_items=10000,bulk_api_crud=True,GROUP=rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE,graceful=False' Test Input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY_AND_PERSIST_TO_ACTIVE', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'graceful': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_20', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_MAJORITY_AND_PERSIST_TO_ACTIVE', 'num_items': '10000', 'bucket_spec': 'single_bucket.default', 'case_number': 20, 'bulk_api_crud': 'True'} test_Orchestrator_Node_failover (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 23:26:18,071 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #20 test_Orchestrator_Node_failover ========= 2021-05-17 23:26:18,805 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #20 test_Orchestrator_Node_failover ========= 2021-05-17 23:26:19,267 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1118 type=client open_count=1 channel=[id: 0x9dabc021, 0.0.0.0/0.0.0.0:48673] timeout=300.0>) 2021-05-17 23:26:40,407 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1119 type=client open_count=1 channel=[id: 0x38364f3e, 0.0.0.0/0.0.0.0:40894] timeout=300.0>) 2021-05-17 23:26:42,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d111a type=client open_count=1 channel=[id: 0xd8ad9171, 0.0.0.0/0.0.0.0:37692] timeout=300.0>) 2021-05-17 23:26:44,447 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d111b type=client open_count=1 channel=[id: 0x3a07d3d9, 0.0.0.0/0.0.0.0:56687] timeout=300.0>) 2021-05-17 23:27:08,355 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d111c type=client open_count=1 channel=[id: 0x2b2fe61a, 0.0.0.0/0.0.0.0:45968] timeout=300.0>) 2021-05-17 23:27:10,372 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d111d type=client open_count=1 channel=[id: 0x0a0d0a40, 0.0.0.0/0.0.0.0:49529] timeout=300.0>) 2021-05-17 23:27:12,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d111e type=client open_count=1 channel=[id: 0xccb8ec56, 0.0.0.0/0.0.0.0:48899] timeout=300.0>) 2021-05-17 23:27:36,622 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d111f type=client open_count=1 channel=[id: 0xcde79cc5, 0.0.0.0/0.0.0.0:51581] timeout=300.0>) 2021-05-17 23:27:38,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1120 type=client open_count=1 channel=[id: 0x54a11f45, 0.0.0.0/0.0.0.0:35833] timeout=300.0>) 2021-05-17 23:27:40,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1121 type=client open_count=1 channel=[id: 0x057fd353, 0.0.0.0/0.0.0.0:58912] timeout=300.0>) 2021-05-17 23:28:04,740 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1122 type=client open_count=1 channel=[id: 0x493193c3, 0.0.0.0/0.0.0.0:34787] timeout=300.0>) 2021-05-17 23:28:06,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1123 type=client open_count=1 channel=[id: 0x7690590c, 0.0.0.0/0.0.0.0:41392] timeout=300.0>) 2021-05-17 23:28:08,780 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1124 type=client open_count=1 channel=[id: 0x4f873bd0, 0.0.0.0/0.0.0.0:44987] timeout=300.0>) 2021-05-17 23:28:32,864 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1125 type=client open_count=1 channel=[id: 0x3f6f16da, 0.0.0.0/0.0.0.0:38585] timeout=300.0>) 2021-05-17 23:28:34,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1126 type=client open_count=1 channel=[id: 0xc4987d63, 0.0.0.0/0.0.0.0:42984] timeout=300.0>) 2021-05-17 23:28:36,900 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 23:28:54,359 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 23:29:08,051 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 23:29:08,073 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #20 test_Orchestrator_Node_failover ========= 2021-05-17 23:29:08,134 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #20 test_Orchestrator_Node_failover ========= 2021-05-17 23:29:18,415 | test | INFO | pool-22-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.75641623892 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:29:18,440 | test | INFO | pool-22-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:29:28,519 | test | INFO | pool-22-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:29:28,546 | test | INFO | pool-22-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.131000042 sec 2021-05-17 23:29:28,828 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 23:29:29,049 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 23:29:29,188 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ | yhhBW5-TiRql_-28-872000 | couchbase | 2 | none | 0 | 0 | 19167969280 | 0 | 0 | +-------------------------+-----------+----------+------------+-----+-------+-------------+----------+-----------+ 2021-05-17 23:29:29,188 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 23:29:33,499 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:29:33,505 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 23:30:20,947 | test | INFO | pool-22-thread-9 | [table_view:display:72] Ops trend for bucket 'yhhBW5-TiRql_-28-872000' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 38.412 | ..............*................................*..................*****.......*............*....*****.........*...*****X | 252.037 | | 250.043 | ..................................................................................................X | 250.043 | | 239.238 | .............................................................................................*X | 239.238 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 23:30:21,000 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 59.6123397943 | 6437847040 | 4618424320 | 2089701376 / 7511588864 | 2203 / 4432 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 60.2843380723 | 6437847040 | 4648321024 | 2039754752 / 7511588864 | 2199 / 4425 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 60.339338989 | 6437847040 | 4247527424 | 2538807296 / 9530753024 | 2201 / 4326 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 61.9010316839 | 6437847040 | 4654858240 | 2043416576 / 7511588864 | 2251 / 4528 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 23:30:25,098 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 23:30:42,036 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +-------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +-------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | yhhBW5-TiRql_-28-872000 | couchbase | 2 | none | 0 | 10000 | 19167969280 | 116921008 | 181050505 | +-------------------------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 23:30:42,038 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #20 test_Orchestrator_Node_failover ========= 2021-05-17 23:30:47,204 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 0 2021-05-17 23:30:56,318 | test | INFO | pool-22-thread-17 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.08000016212 seconds to complete 2021-05-17 23:30:56,323 | test | INFO | pool-22-thread-17 | [common_lib:sleep:22] Sleep 8.08000016212 seconds. Reason: Wait after rebalance complete 2021-05-17 23:31:04,551 | test | INFO | pool-22-thread-18 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 11.9766666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 2.55166666667 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.416666666667 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 8.23 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 23:31:04,571 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:31:09,632 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 13.78 2021-05-17 23:31:14,707 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 32.12 2021-05-17 23:31:19,779 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 51.27 2021-05-17 23:31:24,861 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 70.92 2021-05-17 23:31:29,917 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: running, progress: 89.93 Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1227 type=client open_count=1 channel=[id: 0xd989ed4d, 0.0.0.0/0.0.0.0:47431] timeout=300.0>) 2021-05-17 23:31:34,957 | test | ERROR | pool-22-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-17 23:31:44,783 | test | INFO | pool-22-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:31:44,812 | test | ERROR | pool-22-thread-18 | [rest_client:_http_request:748] GET http://172.23.107.126:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.107.126:8091/pools/default with status 0: unknown pool 2021-05-17 23:31:44,894 | test | INFO | pool-22-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 40.3429999352 sec 2021-05-17 23:31:45,029 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.127 2021-05-17 23:31:45,029 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.126 2021-05-17 23:31:49,798 | test | INFO | pool-22-thread-19 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 10.2628201923 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 12.7631206147 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 10.8864852252 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:31:49,819 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:31:54,874 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:31:59,937 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 4.14 2021-05-17 23:32:05,005 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 22.03 2021-05-17 23:32:10,076 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 39.49 2021-05-17 23:32:15,144 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 56.88 2021-05-17 23:32:20,216 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 74.92 2021-05-17 23:32:25,279 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: running, progress: 93.98 2021-05-17 23:32:35,380 | test | INFO | pool-22-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:32:35,411 | test | INFO | pool-22-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 45.614000082 sec 2021-05-17 23:32:35,411 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 1 2021-05-17 23:32:44,631 | test | INFO | pool-22-thread-20 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.07100009918 seconds to complete 2021-05-17 23:32:44,635 | test | INFO | pool-22-thread-20 | [common_lib:sleep:22] Sleep 8.07100009918 seconds. Reason: Wait after rebalance complete 2021-05-17 23:32:52,836 | test | INFO | pool-22-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 0.781666666667 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 12.3966666667 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 21.2516666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 13.2816666667 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 23:32:52,861 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:32:57,921 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 14.11 2021-05-17 23:33:02,989 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 33.39 2021-05-17 23:33:08,059 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 52.71 2021-05-17 23:33:13,128 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 73.27 2021-05-17 23:33:18,198 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: running, progress: 94.49 2021-05-17 23:33:28,279 | test | INFO | pool-22-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:33:28,365 | test | INFO | pool-22-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 35.5280001163 sec 2021-05-17 23:33:28,434 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 23:33:29,483 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 23:33:30,540 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 23:33:31,608 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.126 2021-05-17 23:33:31,608 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.127 2021-05-17 23:33:35,121 | test | INFO | pool-22-thread-22 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 42.2116666667 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.11180372679 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 4.26840447341 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:33:35,141 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:33:40,197 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:33:45,263 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 5.78 2021-05-17 23:33:50,392 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 23.51 2021-05-17 23:33:55,482 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 41.21 2021-05-17 23:34:00,548 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 59.26 2021-05-17 23:34:05,621 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 77.46 2021-05-17 23:34:10,678 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: running, progress: 96.6 2021-05-17 23:34:20,773 | test | INFO | pool-22-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:34:20,802 | test | INFO | pool-22-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 45.6830000877 sec 2021-05-17 23:34:20,802 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 2 2021-05-17 23:34:30,089 | test | INFO | pool-22-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.06799983978 seconds to complete 2021-05-17 23:34:30,095 | test | INFO | pool-22-thread-23 | [common_lib:sleep:22] Sleep 8.06799983978 seconds. Reason: Wait after rebalance complete 2021-05-17 23:34:38,302 | test | INFO | pool-22-thread-24 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 12.7337577919 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 12.6293685316 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.16666666667 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.48 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:34:38,322 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:34:43,388 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 14.14 2021-05-17 23:34:48,450 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 32.92 2021-05-17 23:34:53,525 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 51.37 2021-05-17 23:34:58,592 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 70.22 2021-05-17 23:35:03,648 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: running, progress: 90.73 Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1228 type=client open_count=1 channel=[id: 0x3ac0613f, 0.0.0.0/0.0.0.0:53972] timeout=300.0>) 2021-05-17 23:35:08,687 | test | ERROR | pool-22-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/pools/default/tasks error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d1229 type=client open_count=1 channel=[id: 0xa4648e6f, 0.0.0.0/0.0.0.0:60350] timeout=300.0>) 2021-05-17 23:35:10,697 | test | ERROR | pool-22-thread-24 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/pools/default/tasks error [Errno 111] Connection refused 2021-05-17 23:35:19,378 | test | INFO | pool-22-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:35:19,398 | test | ERROR | pool-22-thread-24 | [rest_client:_http_request:748] GET http://172.23.107.126:8091/pools/default body: headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/json'} error: 404 reason: unknown "unknown pool" auth: Administrator:password http://172.23.107.126:8091/pools/default with status 0: unknown pool 2021-05-17 23:35:19,476 | test | INFO | pool-22-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 41.1749999523 sec 2021-05-17 23:35:19,575 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.127 2021-05-17 23:35:19,575 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.126 2021-05-17 23:35:25,073 | test | INFO | pool-22-thread-25 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 5.67833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 4.90276270762 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 5.13158114031 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:35:25,096 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:35:30,151 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:35:35,220 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 5.39 2021-05-17 23:35:40,293 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 23.01 2021-05-17 23:35:45,365 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 40.74 2021-05-17 23:35:50,441 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 59.42 2021-05-17 23:35:55,513 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 78.48 2021-05-17 23:36:00,586 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: running, progress: 97.66 2021-05-17 23:36:10,674 | test | INFO | pool-22-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:36:10,703 | test | INFO | pool-22-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 45.6300001144 sec 2021-05-17 23:36:10,704 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1059] Cycle 3 2021-05-17 23:36:19,898 | test | INFO | pool-22-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.07399988174 seconds to complete 2021-05-17 23:36:19,904 | test | INFO | pool-22-thread-26 | [common_lib:sleep:22] Sleep 8.07399988174 seconds. Reason: Wait after rebalance complete 2021-05-17 23:36:28,131 | test | INFO | pool-22-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 3.385 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 12.7095763475 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 14.0916666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.55515925265 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:36:28,154 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:36:33,217 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 14.18 2021-05-17 23:36:38,279 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.2 2021-05-17 23:36:43,342 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 52.47 2021-05-17 23:36:48,404 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.79 2021-05-17 23:36:53,473 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: running, progress: 92.17 2021-05-17 23:37:03,553 | test | INFO | pool-22-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:37:03,638 | test | INFO | pool-22-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 35.5089998245 sec 2021-05-17 23:37:03,714 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 23:37:04,765 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 1 seconds. Reason: orchestrator='undefined' 2021-05-17 23:37:05,832 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1094] master node is: 172.23.107.126 2021-05-17 23:37:05,832 | test | INFO | MainThread | [collections_rebalance:test_Orchestrator_Node_failover:1095] Added node is: 172.23.107.127 2021-05-17 23:37:09,535 | test | INFO | pool-22-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 22.8120468674 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 20.9873662455 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 20.5736762279 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 23:37:09,559 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:37:14,612 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:37:19,671 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 6.29 2021-05-17 23:37:24,763 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 23.87 2021-05-17 23:37:29,831 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 42.19 2021-05-17 23:37:34,898 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 61.45 2021-05-17 23:37:39,963 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 79.39 2021-05-17 23:37:45,042 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: running, progress: 98.24 2021-05-17 23:37:55,128 | test | INFO | pool-22-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:37:55,160 | test | INFO | pool-22-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 45.625 sec 2021-05-17 23:37:56,108 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: yhhBW5-TiRql_-28-872000, Active Resident ratio(DGM): 100% 2021-05-17 23:37:56,108 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: yhhBW5-TiRql_-28-872000, Replica Resident ratio(DGM): 100% 2021-05-17 23:38:20,474 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1289999485 seconds to complete 2021-05-17 23:38:20,493 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 731.870s OK During the test, Remote Connections: 62, Disconnections: 62 SDK Connections: 1, Disconnections: 1 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 20 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_20 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_21 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_21', 'nodes_init': '3', 'GROUP': 'rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 21, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 23:38:29,921 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #21 test_data_load_collections_with_rebalance_in ========= 2021-05-17 23:38:30,604 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #21 test_data_load_collections_with_rebalance_in ========= 2021-05-17 23:38:31,059 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d122a type=client open_count=1 channel=[id: 0xbf02b35b, 0.0.0.0/0.0.0.0:49461] timeout=300.0>) 2021-05-17 23:38:51,862 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d122b type=client open_count=1 channel=[id: 0xa3f77932, 0.0.0.0/0.0.0.0:44606] timeout=300.0>) 2021-05-17 23:38:53,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5d122c type=client open_count=1 channel=[id: 0x5ce6a92a, 0.0.0.0/0.0.0.0:53636] timeout=300.0>) 2021-05-17 23:38:55,898 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d122d type=client open_count=1 channel=[id: 0xccdedd3c, 0.0.0.0/0.0.0.0:40329] timeout=300.0>) 2021-05-17 23:39:20,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d122e type=client open_count=1 channel=[id: 0x9566819b, 0.0.0.0/0.0.0.0:47432] timeout=300.0>) 2021-05-17 23:39:22,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5d122f type=client open_count=1 channel=[id: 0x0c6df062, 0.0.0.0/0.0.0.0:56611] timeout=300.0>) 2021-05-17 23:39:24,913 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1230 type=client open_count=1 channel=[id: 0x469e104a, 0.0.0.0/0.0.0.0:43513] timeout=300.0>) 2021-05-17 23:39:49,005 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1231 type=client open_count=1 channel=[id: 0x1b2f8dae, 0.0.0.0/0.0.0.0:39475] timeout=300.0>) 2021-05-17 23:39:51,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5d1232 type=client open_count=1 channel=[id: 0x2cb0e63f, 0.0.0.0/0.0.0.0:36699] timeout=300.0>) 2021-05-17 23:39:53,042 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1233 type=client open_count=1 channel=[id: 0x8ae50162, 0.0.0.0/0.0.0.0:33994] timeout=300.0>) 2021-05-17 23:40:16,680 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1234 type=client open_count=1 channel=[id: 0xacd16e12, 0.0.0.0/0.0.0.0:56683] timeout=300.0>) 2021-05-17 23:40:18,697 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5d1235 type=client open_count=1 channel=[id: 0xf6b99070, 0.0.0.0/0.0.0.0:42500] timeout=300.0>) 2021-05-17 23:40:20,716 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1236 type=client open_count=1 channel=[id: 0xbf1356a1, 0.0.0.0/0.0.0.0:55666] timeout=300.0>) 2021-05-17 23:40:44,723 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1237 type=client open_count=1 channel=[id: 0x09e466aa, 0.0.0.0/0.0.0.0:44495] timeout=300.0>) 2021-05-17 23:40:46,740 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5d1238 type=client open_count=1 channel=[id: 0x6c82b9e0, 0.0.0.0/0.0.0.0:52444] timeout=300.0>) 2021-05-17 23:40:48,757 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 23:41:06,134 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 23:41:19,825 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 23:41:19,848 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #21 test_data_load_collections_with_rebalance_in ========= 2021-05-17 23:41:19,885 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #21 test_data_load_collections_with_rebalance_in ========= 2021-05-17 23:41:26,667 | test | INFO | pool-23-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 8.9851497525 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 23:41:26,688 | test | INFO | pool-23-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 23:41:36,756 | test | INFO | pool-23-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 23:41:36,798 | test | INFO | pool-23-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1320002079 sec 2021-05-17 23:41:37,048 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 23:42:14,992 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 23:42:15,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 629145600 | 651874888 | 32910755 | | bucket2 | couchbase | 2 | none | 0 | 0 | 629145600 | 607459280 | 59911111 | | default | couchbase | 2 | none | 0 | 0 | 6291456000 | 950254504 | 16603787 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-17 23:42:15,232 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 23:42:42,823 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 23:42:42,832 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 00:05:11,119 | test | INFO | pool-23-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....*..........****X | 5.998 | | 5.298 | ...........*..******.............*..****X | 12.000 | | 3.294 | .......X | 3.294 | | 0.000 | *******************************************........*.................****X | 10.398 | | 6.599 | ...............X | 6.599 | | 4.601 | ..........****X | 4.601 | | 4.499 | ..........*....*****.....*.****X | 8.698 | | 6.699 | ...............X | 6.699 | | 4.500 | ..........*****X | 4.500 | | 1.498 | ..X | 1.498 | | 0.000 | ***********...*........**********X | 4.500 | | 3.499 | .......X | 3.499 | | 3.000 | ......****X | 3.000 | | 0.999 | .X | 0.999 | | 0.000 | ************...*****X | 1.499 | | 0.000 | *************************.*****.*******.****X | 1.500 | | 1.000 | .X | 1.000 | | 0.000 | ******************************...*........*****X | 4.500 | | 0.500 | ******..X | 1.500 | | 1.000 | .****X | 1.000 | | 0.000 | ************...........*****X | 4.501 | | 1.501 | ..X | 1.501 | | 0.000 | ***********.*..****X | 1.500 | | 1.000 | .X | 1.000 | | 0.000 | *****....*...****X | 3.000 | | 2.000 | ...X | 2.000 | | 0.600 | *****.X | 1.100 | | 0.900 | .*****X | 0.900 | | 0.000 | *******************.******........****X | 3.999 | | 2.000 | ...X | 2.000 | | 1.500 | ..***X | 1.500 | | 1.500 | ..*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *****************..*.*****X | 1.500 | | 1.500 | ..******....*....*****....*****X | 6.200 | | 5.600 | ............*****X | 5.601 | | 1.800 | ...X | 1.800 | | 0.700 | *****..X | 1.300 | | 0.800 | ****X | 0.800 | | 0.000 | ************....*...****X | 3.000 | | 2.000 | ...X | 2.000 | | 0.000 | *****************..*....****X | 2.800 | | 2.200 | ....X | 2.200 | | 0.200 | ******...........*****X | 4.500 | | 1.500 | ..X | 1.500 | | 0.000 | ************...........******X | 4.502 | | 4.501 | ..........*****..............*........****X | 13.500 | | 10.500 | ........................X | 10.500 | | 4.499 | ..........****X | 4.499 | | 1.500 | ..X | 1.500 | | 0.000 | ******************....*...****X | 2.999 | | 0.000 | ************......*...******...*****X | 5.000 | | 0.000 | ************...************.................................................................................................*****X | 40.900 | | 6.199 | ..............*****X | 6.199 | | 4.500 | ..........X | 4.500 | | 0.900 | .****X | 0.900 | | 0.000 | ******..............*****X | 5.800 | | 3.200 | ......X | 3.200 | | 1.700 | ...*****.....................*****X | 10.500 | | 0.000 | ************.......*...............*****X | 8.999 | | 1.500 | ..******.....*...****X | 4.500 | | 1.500 | ..X | 1.500 | | 0.000 | ******.....*****X | 2.400 | | 0.600 | *****X | 0.600 | | 0.500 | X | 0.500 | | 0.000 | *****....*...***********X | 3.000 | | 0.000 | ************...*........****X | 4.499 | | 2.999 | ......X | 2.999 | | 0.000 | ***********.........*.....****X | 6.001 | | 0.000 | ******...........******..............*****X | 10.501 | | 1.500 | ..*****X | 1.500 | | 1.500 | ..*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ************........******.X | 4.001 | | 4.000 | ........*****X | 4.000 | | 1.000 | .X | 1.000 | | 0.000 | *****************..*.****X | 1.500 | | 1.000 | .X | 1.000 | | 0.000 | **************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 00:05:12,132 | test | INFO | pool-23-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+--------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+--------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ********...............*................*****........................................................*.............****X | 63.104 | | 51.403 | ................................................................................X | 51.403 | | 32.394 | ..................................................****X | 32.394 | | 21.381 | ................................X | 21.381 | | 0.000 | *******************************************...*........****X | 7.498 | | 6.998 | ..........*******....*****....*.........****X | 18.195 | | 13.795 | ....................X | 13.795 | | 2.801 | ...****X | 2.801 | | 1.499 | .X | 1.499 | | 0.000 | ******.........*****X | 5.999 | | 5.599 | .......*****X | 5.599 | | 3.599 | ....X | 3.599 | | 0.400 | *****....*...****X | 4.500 | | 3.499 | ....X | 3.499 | | 3.000 | ...*****X | 3.000 | | 0.000 | *************************..*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ***********.*****X | 1.000 | | 0.000 | *****.*...****X | 3.000 | | 2.500 | ..X | 2.500 | | 1.500 | .****X | 1.500 | | 1.000 | X | 1.000 | | 0.000 | *****.*.*****X | 1.500 | | 0.000 | ******....*****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | ************************..*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ******..****X | 1.500 | | 1.000 | X | 1.000 | | 0.000 | ************....******........*****X | 7.801 | | 0.000 | *******************************.*.***X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *************************....*...*****X | 4.499 | | 0.000 | *******************..******.*.****X | 3.000 | | 0.000 | ************************...*.****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | ************..........*****X | 6.800 | | 4.700 | ......X | 4.700 | | 0.700 | *****...*...*****X | 4.500 | | 0.000 | ************.*.*****.*....****X | 4.501 | | 3.000 | ...X | 3.000 | | 0.000 | *****.*.****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ************************.*****X | 1.100 | | 0.100 | X | 0.100 | | 0.000 | ******.....******...*****X | 5.500 | | 0.000 | ******...*.****X | 3.000 | | 1.500 | .******.....******....*****X | 7.400 | | 3.600 | ....X | 3.600 | | 1.600 | .****X | 1.600 | | 0.500 | X | 0.500 | | 0.000 | ******..*****X | 1.500 | | 1.500 | .*****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | *****..*****X | 1.500 | | 0.000 | *************..****X | 1.500 | | 0.500 | X | 0.500 | | 0.000 | ******..*****X | 1.500 | | 0.000 | ******.*...*****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | ***********************.*****X | 1.000 | | 0.000 | ******...*.****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | ************..*****X | 1.500 | | 0.000 | ******...*......****X | 5.999 | | 3.999 | .....X | 3.999 | | 3.000 | ...****X | 3.000 | | 1.000 | X | 1.000 | | 0.000 | *******************************........*****X | 5.600 | | 0.000 | *******************.*.****X | 1.500 | | 0.000 | ***************************X | 0.000 | +--------+--------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 00:05:13,273 | test | INFO | pool-23-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....****.....................*.................................****X | 330.842 | | 321.552 | ........................................................*.******.............*.......****X | 438.704 | | 436.110 | ............................................................................X | 436.110 | | 426.043 | ..........................................................................*****.....*.*****X | 458.617 | | 396.606 | .....................................................................X | 396.606 | | 394.125 | .....................................................................***********........*....*****...*.....*****..*.......****X | 561.756 | | 498.402 | .......................................................................................X | 498.402 | | 359.815 | ...............................................................*****...*......****X | 414.858 | | 398.120 | .....................................................................X | 398.120 | | 356.929 | ..............................................................*****X | 356.929 | | 303.982 | .....................................................X | 303.982 | | 272.229 | ...............................................*****.........*.....*****.........*..****X | 414.837 | | 384.931 | ...................................................................X | 384.931 | | 346.438 | ............................................................******...************.....*...****X | 405.957 | | 380.626 | ..................................................................X | 380.626 | | 363.865 | ...............................................................*****...*......****X | 415.065 | | 383.371 | ...................................................................X | 383.371 | | 363.458 | ...............................................................*****.....*.******...........*****X | 458.482 | | 406.698 | .......................................................................X | 406.698 | | 370.315 | ................................................................*****...*.*****............*.....****X | 486.154 | | 458.914 | ................................................................................X | 458.914 | | 449.636 | ..............................................................................****X | 449.636 | | 427.672 | ...........................................................................**********X | 431.527 | | 381.897 | ..................................................................X | 381.897 | | 354.549 | ..............................................................*****...*..****X | 384.702 | | 373.602 | .................................................................X | 373.602 | | 373.271 | .................................................................*****...X | 389.397 | | 388.110 | ....................................................................*****X | 388.110 | | 308.004 | .....................................................******...X | 323.978 | | 311.165 | ......................................................*****...*.....*****.......*****X | 393.913 | | 355.080 | ..............................................................******.*....****X | 387.678 | | 363.286 | ...............................................................X | 363.286 | | 348.787 | .............................................................******........*****X | 399.029 | | 374.171 | .................................................................******....*****X | 394.775 | | 384.575 | ...................................................................X | 384.575 | | 377.799 | ..................................................................****X | 377.799 | | 358.731 | ..............................................................X | 358.731 | | 338.913 | ...........................................................*****....*....*****X | 385.774 | | 335.479 | ..........................................................X | 335.479 | | 315.596 | .......................................................*****......*****X | 351.055 | | 345.711 | ............................................................*****X | 345.711 | | 339.811 | ...........................................................*****X | 342.100 | | 341.895 | ...........................................................*****..X | 348.683 | | 328.451 | .........................................................*****...............*****X | 412.784 | | 382.505 | ...................................................................*****X | 382.505 | | 380.337 | ..................................................................***X | 380.337 | | 353.388 | .............................................................*****X | 353.388 | | 329.515 | .........................................................******..........*...****X | 402.390 | | 386.776 | ...................................................................X | 386.776 | | 372.017 | .................................................................****X | 372.017 | | 360.212 | ...............................................................*.*****..*****X | 376.851 | | 323.472 | ........................................................X | 323.472 | | 294.441 | ...................................................*****...................*****X | 399.312 | | 370.787 | ................................................................*****.X | 376.488 | | 374.588 | .................................................................*****X | 374.588 | | 360.274 | ...............................................................*****X | 360.274 | | 331.157 | .........................................................******.........******..*.*****..X | 405.531 | | 366.299 | ................................................................*****.*.*****X | 381.549 | | 321.333 | ........................................................******..*........****X | 376.719 | | 329.291 | .........................................................X | 329.291 | | 293.455 | ...................................................*****....*...****X | 331.988 | | 327.020 | .........................................................X | 327.020 | | 325.654 | ........................................................****X | 325.654 | | 309.792 | ......................................................X | 309.792 | | 306.813 | .....................................................****X | 306.813 | | 295.860 | ...................................................******........*..****X | 354.100 | | 320.268 | .......................................................*.*****....*....****X | 370.512 | | 362.577 | ...............................................................X | 362.577 | | 346.631 | ............................................................******.........******.......*****X | 437.206 | | 393.199 | ....................................................................*****X | 393.199 | | 384.399 | ...................................................................X | 384.399 | | 351.220 | .............................................................*****...*****X | 368.299 | | 349.524 | .............................................................X | 349.524 | | 342.800 | ...........................................................****X | 342.800 | | 291.355 | ..................................................X | 291.355 | | 277.055 | ................................................******...........*......****X | 374.626 | | 246.897 | ..........................................******....................*****X | 355.739 | | 350.688 | .............................................................*****X | 350.688 | | 321.785 | ........................................................*****X | 321.785 | | 299.291 | ....................................................******..........*..****X | 368.164 | | 322.578 | ........................................................X | 322.578 | | 313.190 | ......................................................*****..........*...****X | 383.188 | | 333.203 | ..........................................................*****X | 333.203 | | 309.171 | ......................................................******..........*****X | 366.653 | | 354.276 | ..............................................................X | 354.276 | | 342.886 | ............................................................******...X | 360.486 | | 347.900 | ............................................................****X | 347.900 | | 343.009 | ............................................................******...X | 363.897 | | 359.511 | ..............................................................*****......*..****X | 403.900 | | 351.054 | .............................................................X | 351.054 | | 329.690 | .........................................................****X | 329.690 | | 324.233 | ........................................................X | 324.233 | | 313.700 | ......................................................*****.....*............****X | 406.727 | | 378.503 | ..................................................................X | 378.503 | | 358.188 | ..............................................................****X | 358.188 | | 350.776 | .............................................................*.*****..........*..****X | 426.657 | | 421.558 | .........................................................................X | 421.558 | | 415.915 | ........................................................................*****X | 415.915 | | 385.736 | ...................................................................*****X | 385.736 | | 383.623 | ...................................................................*****X | 383.623 | | 327.344 | .........................................................******.******.......................*****X | 461.121 | | 433.991 | ............................................................................X | 433.991 | | 357.550 | ..............................................................****X | 357.550 | | 351.251 | .............................................................******...X | 369.272 | | 348.819 | .............................................................*****X | 351.596 | | 346.847 | ............................................................***********X | 348.286 | | 330.600 | .........................................................******.........*****X | 376.697 | | 348.303 | ............................................................X | 348.303 | | 342.778 | ...........................................................****X | 342.778 | | 329.733 | .........................................................X | 329.733 | | 324.143 | ........................................................******...............******..........*****X | 465.797 | | 460.133 | ................................................................................******..........*.........*****X | 562.078 | | 454.539 | ...............................................................................***X | 454.539 | +---------+---------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 00:05:13,571 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 73.3333333333 | 6437847040 | 4006002688 | 2753388544 / 7511588864 | 168195 / 335993 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 80.2359882006 | 6437847040 | 4061728768 | 2671362048 / 7511588864 | 168308 / 336331 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 62.9463987718 | 6437847040 | 3732000768 | 3117588480 / 9530753024 | 167879 / 336474 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 00:05:22,450 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 00:06:17,197 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 165011800 | 247933991 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 161075504 | 265147124 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 438765336 | 436655259 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 00:06:17,197 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #21 test_data_load_collections_with_rebalance_in ========= 2021-05-18 00:06:23,513 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in 2021-05-18 00:06:23,515 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-18 00:06:23,575 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 00:06:31,937 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 00:06:33,911 | test | INFO | pool-23-thread-1 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 16.4336072268 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 15.055913871 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 25.6766666667 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 00:06:33,980 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 00:06:39,792 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.07 2021-05-18 00:06:45,086 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.74 2021-05-18 00:06:50,217 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.43 2021-05-18 00:06:55,358 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 8.99 2021-05-18 00:07:00,484 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 12.45 2021-05-18 00:07:05,607 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 15.37 2021-05-18 00:07:10,717 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 18.8 2021-05-18 00:07:13,676 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:13,677 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-10 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-35 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-140 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:15,861 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 21.75 2021-05-18 00:07:21,032 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 25.01 2021-05-18 00:07:26,194 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 28.16 2021-05-18 00:07:27,476 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-555 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:27,477 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-555 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-473 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:31,335 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.1 2021-05-18 00:07:36,499 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 32.86 2021-05-18 00:07:38,849 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:38,849 | test | INFO | pool-23-thread-4 | [table_view:display:72] Keys failed in bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-947 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:07:41,625 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.53 2021-05-18 00:07:46,763 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 36.6 2021-05-18 00:07:51,888 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 40.05 2021-05-18 00:07:57,016 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 43.81 2021-05-18 00:08:02,183 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 47.7 2021-05-18 00:08:07,322 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 51.27 2021-05-18 00:08:12,460 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 54.46 2021-05-18 00:08:15,944 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in bucket2:4uEGTjKSHL-JLZ1D-28-130000:4aklOun1bzNZ6I7A_w6-28-536000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-94 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-107 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:15,946 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in bucket2:4uEGTjKSHL-JLZ1D-28-130000:4aklOun1bzNZ6I7A_w6-28-536000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-94 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-107 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:16,908 | test | INFO | pool-23-thread-17 | [table_view:display:72] Keys failed in bucket2:scope1:mPh8diaRI9VL-37-101000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:17,701 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 57.27 2021-05-18 00:08:22,845 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.13 2021-05-18 00:08:28,015 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.89 2021-05-18 00:08:33,122 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 65.31 2021-05-18 00:08:37,186 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in bucket2:4uEGTjKSHL-JLZ1D-28-130000:4aklOun1bzNZ6I7A_w6-28-536000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-756 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-694 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:37,187 | test | INFO | pool-23-thread-18 | [table_view:display:72] Keys failed in bucket2:4uEGTjKSHL-JLZ1D-28-130000:4aklOun1bzNZ6I7A_w6-28-536000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-756 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-694 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:38,214 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 00:08:43,338 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.57 2021-05-18 00:08:47,263 | test | INFO | pool-23-thread-11 | [table_view:display:72] Keys failed in default:_default:DByf5See-37-130000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:48,474 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.85 2021-05-18 00:08:49,088 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-58 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-54 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-67 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-104 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-97 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-125 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:49,240 | test | INFO | pool-23-thread-6 | [table_view:display:72] Keys failed in default:scope2:4E-37-137000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-957 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-936 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-973 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:51,490 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in default:scope2:pDj3K8QozHRUf2Q-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-450 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:52,970 | test | INFO | pool-23-thread-8 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-547 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-520 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:52,973 | test | INFO | pool-23-thread-14 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-502 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-676 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:08:53,605 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.0 2021-05-18 00:08:58,749 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.77 2021-05-18 00:09:03,891 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 77.8 2021-05-18 00:09:04,969 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:scope1:BQ2LS-37-134000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-808 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-805 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-746 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-844 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-734 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-753 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-742 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-721 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-782 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-882 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:09,017 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 80.03 2021-05-18 00:09:14,164 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 82.44 2021-05-18 00:09:15,670 | test | INFO | pool-23-thread-29 | [table_view:display:72] Keys failed in default:scope2:collection2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-825 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-723 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-786 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-897 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-860 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:17,700 | test | INFO | pool-23-thread-11 | [table_view:display:72] Keys failed in default:_default:DByf5See-37-130000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-839 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-825 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-878 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-898 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-723 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-786 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-897 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-771 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-860 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:19,276 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 84.78 2021-05-18 00:09:20,596 | test | INFO | pool-23-thread-22 | [table_view:display:72] Keys failed in default:_default:Azt_WVTOXFpjT6K-vu7-37-129000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-605 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-549 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-623 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-668 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-566 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-640 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-681 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-517 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-616 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-615 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-632 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-533 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-677 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-575 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-553 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-653 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-631 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-672 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-650 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-552 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-693 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-591 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-592 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:23,931 | test | INFO | pool-23-thread-23 | [table_view:display:72] Keys failed in default:scope2:yr8TfIij5m-37-136000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-337 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-248 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-322 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-211 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:24,381 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 86.6 2021-05-18 00:09:25,938 | test | INFO | pool-23-thread-20 | [table_view:display:72] Keys failed in default:scope2:POrYD-37-135000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-724 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-889 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-703 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-824 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-843 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-888 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-789 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-723 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-764 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-861 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-884 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-860 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-839 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-811 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-795 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-772 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:27,423 | test | INFO | pool-23-thread-19 | [table_view:display:72] Keys failed in default:scope1:BQ2LS-37-134000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-14 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-15 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-11 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-77 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-99 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-56 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-13 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-51 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-73 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-105 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-186 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-187 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-182 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-46 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-118 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-64 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-114 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-151 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-190 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:29,107 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:scope1:BQ2LS-37-134000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-959 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-935 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-914 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-943 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-952 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:29,579 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 88.17 2021-05-18 00:09:34,763 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 89.74 2021-05-18 00:09:35,625 | test | INFO | pool-23-thread-29 | [table_view:display:72] Keys failed in default:scope2:collection2 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-906 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-926 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-916 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-902 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-979 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-969 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-951 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:37,351 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-207 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-306 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-249 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-302 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-347 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-389 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-221 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-321 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-384 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-220 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-341 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-280 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-219 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-238 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-214 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-355 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-397 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-310 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-296 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-292 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-370 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:37,371 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-645 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-536 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-512 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-675 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-587 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-697 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:38,961 | test | INFO | pool-23-thread-8 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-707 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-806 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-869 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-748 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-705 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-887 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-745 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-710 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-776 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-842 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-894 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-780 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-880 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:39,875 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 91.14 2021-05-18 00:09:40,154 | test | INFO | pool-23-thread-10 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-74 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-149 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-194 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-191 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:40,226 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-507 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-658 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-579 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-667 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-568 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-536 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-610 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-622 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-586 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-573 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-695 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-680 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-590 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:42,884 | test | INFO | pool-23-thread-3 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-18 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-88 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-61 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-74 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-138 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-149 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-42 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-162 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-194 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-191 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:44,994 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 92.5 2021-05-18 00:09:45,607 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:vC%PoSwWpBM-31-707000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1039 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:45,608 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:vC%PoSwWpBM-31-707000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1039 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:45,868 | test | INFO | pool-23-thread-19 | [table_view:display:72] Keys failed in default:scope1:BQ2LS-37-134000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-227 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-216 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-205 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-228 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-386 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-253 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:48,415 | test | INFO | pool-23-thread-23 | [table_view:display:72] Keys failed in default:scope2:yr8TfIij5m-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-438 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-479 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-431 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:49,785 | test | INFO | pool-23-thread-14 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-829 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-839 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-847 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-737 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-715 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-812 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-724 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-702 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-889 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-788 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-854 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-811 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-723 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-720 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-765 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-795 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-785 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-760 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-860 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:50,135 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 93.59 2021-05-18 00:09:53,907 | test | INFO | pool-23-thread-11 | [table_view:display:72] Keys failed in default:_default:DByf5See-37-130000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-967 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-960 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:55,275 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 94.81 2021-05-18 00:09:56,532 | test | INFO | pool-23-thread-20 | [table_view:display:72] Keys failed in default:scope2:POrYD-37-135000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-932 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-960 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-972 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:56,533 | test | INFO | pool-23-thread-22 | [table_view:display:72] Keys failed in default:_default:Azt_WVTOXFpjT6K-vu7-37-129000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-841 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:09:57,201 | test | INFO | pool-23-thread-10 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-239 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-247 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-377 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-202 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-240 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-392 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:00,357 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 95.66 2021-05-18 00:10:01,999 | test | INFO | pool-23-thread-8 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-940 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:02,496 | test | INFO | pool-23-thread-14 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-937 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-940 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:02,611 | test | INFO | pool-23-thread-3 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-239 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-324 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-210 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-255 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-380 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:02,611 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-804 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-850 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:03,538 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:scope2:qTdtMsCki0xtI%-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-405 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-428 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-478 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-413 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-440 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:05,459 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 96.26 2021-05-18 00:10:07,213 | test | INFO | pool-23-thread-22 | [table_view:display:72] Keys failed in default:_default:Azt_WVTOXFpjT6K-vu7-37-129000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-900 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-964 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-961 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:07,859 | test | INFO | pool-23-thread-19 | [table_view:display:72] Keys failed in default:scope1:BQ2LS-37-134000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-485 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-480 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:10,555 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 97.39 2021-05-18 00:10:13,591 | test | INFO | pool-23-thread-28 | [table_view:display:72] Keys failed in default:scope1:NQNQhaAK1-37-133000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-624 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-635 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-589 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-670 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:14,302 | test | INFO | pool-23-thread-3 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-407 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:14,302 | test | INFO | pool-23-thread-10 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-425 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-460 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-480 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:14,305 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-719 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-815 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-846 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-767 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-841 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:15,437 | test | INFO | pool-23-thread-26 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-924 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-947 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:15,650 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 98.48 2021-05-18 00:10:20,803 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 99.39 2021-05-18 00:10:21,934 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:Yz0j-30-688000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:21,934 | test | INFO | pool-23-thread-12 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:Yz0j-30-688000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-407 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-597 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:22,130 | test | INFO | pool-23-thread-22 | [table_view:display:72] Keys failed in default:scope2:Om4-37-137000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-602 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-597 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-598 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:23,592 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:scope2:KZSs1hL4s2W-37-137000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-57 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-80 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:23,716 | test | INFO | pool-23-thread-20 | [table_view:display:72] Keys failed in default:scope2:oDYQlY0uYfeL1mS60-37-136000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-519 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-503 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-558 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-544 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:26,278 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 00:10:26,855 | test | INFO | pool-23-thread-27 | [table_view:display:72] Keys failed in default:MUK8t5s2-37-137000:7-37-139000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-909 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-954 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:29,006 | test | INFO | pool-23-thread-10 | [table_view:display:72] Keys failed in default:fZH4weINYdYu2KP7UCR-37-140000:wdEfjM-37-142000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-613 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:29,252 | test | INFO | pool-23-thread-7 | [table_view:display:72] Keys failed in default:Cxu_-30-947000:qgz1g8738oNLoIDU3ld-31-117000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:29,253 | test | INFO | pool-23-thread-7 | [table_view:display:72] Keys failed in default:Cxu_-30-947000:qgz1g8738oNLoIDU3ld-31-117000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1435 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:29,552 | test | INFO | pool-23-thread-28 | [table_view:display:72] Keys failed in default:scope1:NQNQhaAK1-37-133000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-768 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-856 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-731 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-774 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-881 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-783 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:30,163 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:vC%PoSwWpBM-31-707000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:30,164 | test | INFO | pool-23-thread-15 | [table_view:display:72] Keys failed in default:8T5B%8ULrJgXeMQcAEjt-30-536000:vC%PoSwWpBM-31-707000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1619 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1789 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:32,700 | test | INFO | pool-23-thread-22 | [table_view:display:72] Keys failed in default:scope2:Om4-37-137000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-871 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:34,894 | test | INFO | pool-23-thread-30 | [table_view:display:72] Keys failed in default:scope2:KZSs1hL4s2W-37-137000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-313 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-356 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-283 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:10:36,421 | test | INFO | pool-23-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 00:10:36,479 | test | INFO | pool-23-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 242.566999912 sec 2021-05-18 00:14:40,858 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:14:40,861 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-18 00:14:40,862 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-18 00:14:40,864 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 6 docs from 0 to 5000 2021-05-18 00:14:40,867 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 24 docs from 500 to 980 2021-05-18 00:14:40,868 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 2 docs from 0 to 500 2021-05-18 00:14:57,851 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 35 docs from 500 to 980 2021-05-18 00:14:57,854 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 5 docs from 0 to 5000 2021-05-18 00:14:57,855 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 00:14:57,855 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 27 docs from 0 to 500 2021-05-18 00:14:57,857 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 27 docs from 500 to 980 2021-05-18 00:14:57,857 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 30 docs from 500 to 980 2021-05-18 00:14:57,858 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 30 docs from 500 to 980 2021-05-18 00:14:57,858 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 30 docs from 500 to 980 2021-05-18 00:14:57,859 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 47 docs from 0 to 500 2021-05-18 00:15:47,111 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 40 docs from 500 to 980 2021-05-18 00:15:47,111 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 38 docs from 0 to 500 2021-05-18 00:15:47,112 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 27 docs from 500 to 980 2021-05-18 00:15:47,112 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 25 docs from 500 to 980 2021-05-18 00:15:47,114 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 33 docs from 0 to 500 2021-05-18 00:15:47,114 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 25 docs from 0 to 500 2021-05-18 00:15:47,115 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 16 docs from 500 to 980 2021-05-18 00:15:47,115 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 00:15:47,117 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 8 docs from 500 to 980 2021-05-18 00:15:47,118 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 9 docs from 0 to 500 2021-05-18 00:15:47,118 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-18 00:15:47,119 | test | ERROR | pool-23-thread-25 | [task:call:3809] Failed to load 2 docs from 500 to 980 2021-05-18 00:22:52,342 | test | INFO | pool-23-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ********X | 1.000 | | 0.400 | *********..*X | 3.099 | | 3.099 | .X | 3.099 | | 2.799 | .X | 2.799 | | 0.000 | ********************................*..***.......****.*.****.........****..*.***.*......**X | 53.889 | | 44.736 | .....................................*****......*****.***......*..*********....X | 66.293 | | 65.903 | .......................................................***X | 65.903 | | 58.497 | .................................................X | 58.497 | | 56.297 | ...............................................*****...........*****.......*.............****X | 92.685 | | 76.862 | .................................................................X | 76.862 | | 72.488 | .............................................................****....*...****X | 81.311 | | 73.287 | ..............................................................X | 73.287 | | 72.974 | .............................................................*****....****X | 77.207 | | 75.691 | ................................................................*...****X | 79.343 | | 77.643 | .................................................................*.....****X | 83.527 | | 61.803 | ....................................................****X | 61.803 | | 61.202 | ...................................................X | 61.202 | | 57.688 | ................................................*****..........*****X | 69.390 | | 57.789 | ................................................******.....****X | 63.787 | | 62.500 | ....................................................X | 62.500 | | 58.704 | .................................................****...........*...........****X | 84.789 | | 78.291 | ..................................................................****X | 78.291 | | 74.896 | ...............................................................X | 74.896 | | 63.985 | ......................................................*****..*........****X | 76.297 | | 59.681 | ..................................................****X | 59.681 | | 58.980 | .................................................*..*****..*****......................*........*.....***.....*......***X | 116.278 | | 111.270 | ..............................................................................................X | 111.270 | | 88.293 | ..........................................................................****X | 88.293 | | 83.987 | .......................................................................X | 83.987 | | 79.896 | ...................................................................****..........X | 91.806 | | 91.101 | .............................................................................*****......*****.......*..***X | 108.779 | | 93.293 | ...............................................................................X | 93.293 | | 84.706 | .......................................................................****X | 84.706 | | 72.691 | .............................................................****X | 72.691 | | 71.291 | ............................................................*.....****X | 77.082 | | 68.383 | .........................................................X | 68.383 | | 11.706 | .........*****...........................................*****...*......*****.....X | 78.901 | | 77.005 | .................................................................***X | 77.005 | | 73.101 | .............................................................X | 73.101 | | 71.695 | ............................................................***X | 71.695 | | 69.562 | ..........................................................*.****....X | 75.091 | | 74.527 | ...............................................................*****.....****X | 80.896 | | 33.579 | ...........................X | 33.579 | | 3.199 | .****X | 3.199 | | 2.800 | .****X | 2.800 | | 2.299 | X | 2.299 | | 1.900 | ****X | 1.900 | | 1.199 | X | 1.199 | | 0.800 | ****.******.*.***X | 3.699 | | 1.400 | ****X | 1.400 | | 1.000 | X | 1.000 | | 0.600 | ****X | 0.600 | | 0.500 | X | 0.500 | | 0.300 | *****.*.*****.****X | 4.401 | | 3.401 | .****X | 3.401 | | 3.301 | .X | 3.301 | | 1.500 | *****..****X | 3.900 | | 3.700 | ..X | 3.700 | | 3.000 | .****X | 3.000 | | 2.700 | .****X | 2.801 | | 2.701 | .X | 2.701 | | 0.600 | **********.*.****X | 2.900 | | 1.300 | ****************X | 2.100 | | 0.700 | *****.*****X | 2.100 | | 1.700 | X | 1.700 | | 1.400 | ****X | 1.400 | | 0.700 | ****X | 0.700 | | 0.300 | *****..*.****X | 4.200 | | 3.302 | .X | 3.302 | | 2.901 | .****.X | 3.900 | | 3.400 | .*****.****X | 3.498 | | 3.298 | .X | 3.298 | | 2.899 | .X | 2.899 | | 2.301 | ***X | 2.301 | | 1.600 | ******..*****X | 4.600 | | 3.200 | .****X | 3.200 | | 1.900 | **X | 1.900 | +---------+--------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 00:22:52,459 | test | INFO | pool-23-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****X | 0.200 | | 0.000 | ********.*.***.................****........****..*..**X | 38.094 | | 37.666 | ..............................****..X | 40.526 | | 39.031 | ...............................***...................*.....***..X | 70.384 | | 68.384 | .......................................................***X | 68.384 | | 66.736 | ......................................................****............*............***.....*..........***X | 114.202 | | 79.664 | ................................................................***X | 79.664 | | 76.484 | ..............................................................X | 76.484 | | 76.085 | .............................................................**X | 76.085 | | 72.479 | ...........................................................X | 72.479 | | 69.021 | ........................................................***X | 69.021 | | 65.762 | .....................................................****...............*.........***X | 94.384 | | 82.288 | ...................................................................****X | 82.288 | | 81.291 | ..................................................................*.......***X | 89.866 | | 80.306 | .................................................................*****........****X | 90.495 | | 75.597 | .............................................................X | 75.597 | | 64.801 | ....................................................*****........................................*****...*...****X | 120.781 | | 101.768 | ...................................................................................X | 101.768 | | 90.701 | ..........................................................................****.......*..****X | 101.764 | | 73.890 | ............................................................X | 73.890 | | 69.719 | ........................................................****X | 69.719 | | 63.450 | ...................................................*****..*..*****...***********..*...****X | 77.388 | | 69.484 | ........................................................******...........*****......*................****X | 108.817 | | 104.720 | .....................................................................................X | 104.720 | | 80.386 | .................................................................****X | 80.386 | | 69.291 | ........................................................****X | 69.291 | | 66.084 | .....................................................*****.******..*.*****.........******.*...***X | 86.379 | | 35.053 | ............................X | 35.053 | | 17.406 | .............***X | 17.406 | | 8.695 | ......X | 8.695 | | 0.399 | ****X | 0.399 | | 0.299 | X | 0.299 | | 0.200 | ****.****X | 2.099 | | 0.998 | X | 0.998 | | 0.000 | ***********.*****.***X | 2.599 | | 2.199 | X | 2.199 | | 0.900 | ****X | 0.900 | | 0.899 | ******.*.***X | 2.699 | | 1.998 | X | 1.998 | | 1.799 | ***X | 1.799 | | 1.700 | X | 1.700 | | 1.600 | ****X | 1.600 | | 0.400 | *****.*.***X | 2.498 | | 1.598 | X | 1.598 | | 0.800 | ****.*****X | 1.801 | | 1.701 | X | 1.701 | | 0.700 | *****.************.****X | 2.600 | | 1.199 | X | 1.199 | | 0.100 | *********.******...****X | 5.599 | | 3.399 | .X | 3.399 | | 1.300 | ****X | 1.300 | | 0.600 | *****X | 1.000 | | 0.800 | ***X | 0.800 | | 0.500 | ******.******..****X | 3.700 | | 3.201 | .X | 3.201 | | 3.100 | .****X | 3.100 | | 2.100 | X | 2.100 | | 1.200 | *****.**********.*.*****X | 4.000 | | 2.500 | .***X | 2.500 | | 2.200 | X | 2.200 | | 1.300 | X | 1.300 | | 0.600 | *****.****.*****X | 3.000 | | 1.800 | X | 1.800 | | 1.400 | ****.****X | 2.800 | | 2.800 | .X | 2.800 | | 2.601 | .****X | 2.601 | | 2.501 | .X | 2.501 | | 1.900 | *****.****X | 2.599 | | 1.700 | X | 1.700 | | 1.000 | ***********....****X | 5.801 | | 2.900 | .X | 2.900 | | 1.499 | *****..*****X | 4.201 | | 2.800 | .*****X | 3.400 | | 3.300 | .****X | 3.300 | | 3.200 | .X | 3.200 | | 3.000 | .****X | 3.000 | | 2.100 | X | 2.100 | | 1.500 | *X | 1.500 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 00:22:53,927 | test | INFO | pool-23-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *...........................****......................................................................***X | 734.090 | | 248.288 | ................................X | 248.288 | | 32.797 | ...***......................****.....................................***X | 475.311 | | 235.737 | ..............................***X | 235.737 | | 211.801 | ...........................***X | 211.801 | | 88.984 | ..........***X | 88.984 | | 69.519 | ........****...*.***...................*........................................X | 544.782 | | 544.782 | .......................................................................**X | 544.782 | | 93.568 | ...........*****..................***X | 231.284 | | 206.678 | ..........................****...........*X | 287.204 | | 287.204 | .....................................*X | 287.204 | | 264.737 | ..................................X | 264.737 | | 112.211 | .............***X | 112.211 | | 70.349 | ........*X | 70.349 | | 70.349 | ........**....*..............*X | 204.653 | | 204.653 | ..........................*X | 204.653 | | 149.433 | ..................X | 149.433 | | 109.893 | .............***X | 109.893 | | 90.623 | ...........X | 90.623 | | 90.623 | ...........***...*...***X | 135.952 | | 130.577 | ................*.****.......................*...............****.******....*.***X | 470.257 | | 462.138 | ............................................................X | 462.138 | | 446.541 | ..........................................................****X | 446.541 | | 324.617 | ..........................................******..****X | 343.061 | | 316.983 | .........................................X | 316.983 | | 189.516 | ........................*****..............*..........****X | 372.139 | | 302.995 | .......................................******.....................****X | 460.568 | | 283.111 | ....................................******......*..***X | 342.222 | | 331.529 | ...........................................X | 331.529 | | 283.092 | ....................................X | 283.092 | | 281.186 | ....................................***X | 281.186 | | 253.107 | ................................X | 253.107 | | 221.752 | ............................****X | 221.752 | | 211.333 | ...........................*****.....................*....****X | 404.331 | | 362.828 | ...............................................****X | 362.828 | | 282.745 | ....................................****X | 282.745 | | 281.828 | ....................................*....***X | 307.979 | | 307.979 | ........................................X | 307.979 | | 205.096 | ..........................X | 205.096 | | 172.462 | ......................****...*..............*....***X | 335.978 | | 321.792 | .........................................X | 321.792 | | 292.944 | ......................................****..........*..*....**X | 413.650 | | 401.302 | ....................................................X | 401.302 | | 307.555 | ........................................***X | 307.555 | | 295.737 | ......................................X | 295.737 | | 273.484 | ...................................*****.*....****X | 307.474 | | 282.509 | ....................................*****..............*...........****X | 466.085 | | 378.629 | .................................................*****...*....***X | 429.210 | | 425.957 | .......................................................*.....****X | 458.302 | | 350.610 | .............................................X | 350.610 | | 312.234 | ........................................****.............*.......*................X | 581.601 | | 581.601 | ............................................................................**X | 581.601 | | 553.407 | ........................................................................X | 553.407 | | 519.561 | ....................................................................***X | 519.561 | | 433.753 | ........................................................X | 433.753 | | 404.681 | .....................................................*****....****X | 437.136 | | 291.362 | .....................................*****......*....***X | 360.087 | | 344.553 | ............................................X | 344.553 | | 334.223 | ...........................................****X | 334.223 | | 301.682 | .......................................*****...........................*....................****X | 652.783 | | 569.524 | ...........................................................................****X | 569.524 | | 550.458 | ........................................................................X | 550.458 | | 507.172 | ..................................................................*****.......*..****X | 571.532 | | 437.886 | .........................................................*****............*...........****X | 612.506 | | 497.628 | .................................................................****X | 497.628 | | 458.228 | ............................................................X | 458.228 | | 446.427 | ..........................................................X | 446.427 | | 446.427 | ..........................................................*X | 446.427 | | 446.427 | ..........................................................*X | 446.427 | | 444.378 | ..........................................................*****..*..***X | 472.226 | | 468.400 | .............................................................*....*****......****X | 544.680 | | 494.370 | ................................................................X | 494.370 | | 468.681 | .............................................................****X | 468.681 | | 432.557 | ........................................................*****.............*......****X | 570.876 | | 483.164 | ...............................................................X | 483.164 | | 417.216 | ......................................................**X | 417.216 | | 417.216 | ......................................................X | 417.216 | | 411.369 | .....................................................*..*****....*****.*.****X | 464.963 | | 418.395 | ......................................................*****.......*....****X | 498.462 | | 442.098 | ..........................................................*****...*..****.X | 487.613 | | 467.994 | .............................................................****X | 467.994 | | 452.856 | ...........................................................****X | 452.856 | | 449.491 | ..........................................................X | 449.491 | | 437.951 | .........................................................X | 437.951 | | 408.603 | .....................................................***X | 408.603 | | 375.369 | .................................................X | 375.369 | | 355.798 | ..............................................*****..........******................*****.*....****X | 587.824 | | 558.079 | .........................................................................****X | 558.079 | | 528.924 | .....................................................................X | 528.924 | | 518.540 | ....................................................................*****..X | 534.450 | | 532.154 | ......................................................................*****...*****..........*........*****........***X | 749.170 | +---------+-------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 00:22:53,980 | test | WARNING | Thread-48594 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:4uEGTjKSHL-JLZ1D-28-130000:4aklOun1bzNZ6I7A_w6-28-536000 create failed keys from task: [u'test_collections-704', u'test_collections-94', u'test_collections-107', u'test_collections-756', u'test_collections-694'] 2021-05-18 00:22:54,015 | test | WARNING | Thread-48653 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:scope1:mPh8diaRI9VL-37-101000 update failed keys from task: [u'test_collections-4'] 2021-05-18 00:22:54,025 | test | WARNING | Thread-48749 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rB%Euu3ayOizh-28-964000:JHMTsUpuJZF9mJ-29-322000 create failed keys from task: [u'test_collections-10', u'test_collections-947', u'test_collections-35', u'test_collections-555', u'test_collections-473', u'test_collections-140'] 2021-05-18 00:22:54,065 | test | WARNING | Thread-48861 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:DByf5See-37-130000 update failed keys from task: [u'test_collections-729', u'test_collections-847', u'test_collections-922', u'test_collections-967', u'test_collections-723', u'test_collections-720', u'test_collections-765', u'test_collections-960', u'test_collections-939', u'test_collections-737', u'test_collections-812', u'test_collections-977', u'test_collections-898', u'test_collections-854', u'test_collections-657', u'test_collections-897', u'test_collections-771', u'test_collections-829', u'test_collections-825', u'test_collections-702', u'test_collections-788', u'test_collections-786', u'test_collections-860', u'test_collections-839', u'test_collections-837', u'test_collections-878', u'test_collections-711', u'test_collections-676', u'test_collections-754', u'test_collections-872'] 2021-05-18 00:22:54,079 | test | WARNING | Thread-48868 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:Azt_WVTOXFpjT6K-vu7-37-129000 update failed keys from task: [u'test_collections-605', u'test_collections-846', u'test_collections-964', u'test_collections-767', u'test_collections-921', u'test_collections-566', u'test_collections-841', u'test_collections-961', u'test_collections-640', u'test_collections-681', u'test_collections-618', u'test_collections-616', u'test_collections-815', u'test_collections-615', u'test_collections-533', u'test_collections-578', u'test_collections-575', u'test_collections-653', u'test_collections-650', u'test_collections-692', u'test_collections-693', u'test_collections-508', u'test_collections-900', u'test_collections-549', u'test_collections-868', u'test_collections-623', u'test_collections-668', u'test_collections-784', u'test_collections-719', u'test_collections-838', u'test_collections-517', u'test_collections-632', u'test_collections-677', u'test_collections-553', u'test_collections-631', u'test_collections-672', u'test_collections-552', u'test_collections-593', u'test_collections-591', u'test_collections-592'] 2021-05-18 00:22:54,092 | test | WARNING | Thread-48923 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:MUK8t5s2-37-137000:7-37-139000 update failed keys from task: [u'test_collections-846', u'test_collections-568', u'test_collections-767', u'test_collections-841', u'test_collections-680', u'test_collections-815', u'test_collections-658', u'test_collections-579', u'test_collections-536', u'test_collections-610', u'test_collections-573', u'test_collections-695', u'test_collections-909', u'test_collections-507', u'test_collections-868', u'test_collections-667', u'test_collections-622', u'test_collections-586', u'test_collections-784', u'test_collections-719', u'test_collections-838', u'test_collections-954', u'test_collections-555', u'test_collections-673', u'test_collections-590'] 2021-05-18 00:22:54,094 | test | WARNING | Thread-48935 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Cxu_-30-947000:qgz1g8738oNLoIDU3ld-31-117000 create failed keys from task: [u'test_collections-1435'] 2021-05-18 00:22:54,095 | test | WARNING | Thread-48940 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:fZH4weINYdYu2KP7UCR-37-140000:wdEfjM-37-142000 update failed keys from task: [u'test_collections-613', u'test_collections-682'] 2021-05-18 00:22:54,098 | test | WARNING | Thread-48951 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 update failed keys from task: [u'test_collections-528', u'test_collections-924', u'test_collections-804', u'test_collections-846', u'test_collections-645', u'test_collections-568', u'test_collections-767', u'test_collections-921', u'test_collections-680', u'test_collections-815', u'test_collections-658', u'test_collections-579', u'test_collections-536', u'test_collections-850', u'test_collections-697', u'test_collections-947', u'test_collections-868', u'test_collections-787', u'test_collections-586', u'test_collections-784', u'test_collections-587', u'test_collections-719', u'test_collections-838', u'test_collections-555', u'test_collections-512', u'test_collections-675', u'test_collections-673'] 2021-05-18 00:22:54,101 | test | WARNING | Thread-48974 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:NQNQhaAK1-37-133000 update failed keys from task: [u'test_collections-768', u'test_collections-624', u'test_collections-589', u'test_collections-881', u'test_collections-783', u'test_collections-735', u'test_collections-856', u'test_collections-757', u'test_collections-834', u'test_collections-635', u'test_collections-712', u'test_collections-731', u'test_collections-774', u'test_collections-871', u'test_collections-670', u'test_collections-770'] 2021-05-18 00:22:54,122 | test | WARNING | Thread-49045 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8T5B%8ULrJgXeMQcAEjt-30-536000:Yz0j-30-688000 create failed keys from task: [u'test_collections-407', u'test_collections-597'] 2021-05-18 00:22:54,122 | test | WARNING | Thread-49046 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8T5B%8ULrJgXeMQcAEjt-30-536000:vC%PoSwWpBM-31-707000 create failed keys from task: [u'test_collections-1619', u'test_collections-1039', u'test_collections-1789'] 2021-05-18 00:22:54,135 | test | WARNING | Thread-49034 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:4E-37-137000 update failed keys from task: [u'test_collections-957', u'test_collections-936', u'test_collections-934', u'test_collections-973', u'test_collections-950'] 2021-05-18 00:22:54,153 | test | WARNING | Thread-48976 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:BQ2LS-37-134000 update failed keys from task: [u'test_collections-808', u'test_collections-805', u'test_collections-803', u'test_collections-844', u'test_collections-721', u'test_collections-840', u'test_collections-882', u'test_collections-938', u'test_collections-935', u'test_collections-734', u'test_collections-708', u'test_collections-906', u'test_collections-746', u'test_collections-943', u'test_collections-742', u'test_collections-782', u'test_collections-959', u'test_collections-837', u'test_collections-914', u'test_collections-953', u'test_collections-753', u'test_collections-951', u'test_collections-754', u'test_collections-952'] 2021-05-18 00:22:54,289 | test | WARNING | Thread-49008 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:KZSs1hL4s2W-37-137000 delete failed keys from task: [u'test_collections-98', u'test_collections-57', u'test_collections-159', u'test_collections-313', u'test_collections-80', u'test_collections-356', u'test_collections-283', u'test_collections-170', u'test_collections-192'] 2021-05-18 00:22:54,318 | test | WARNING | Thread-49012 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:qTdtMsCki0xtI%-37-136000 update failed keys from task: [u'test_collections-927', u'test_collections-847', u'test_collections-724', u'test_collections-889', u'test_collections-723', u'test_collections-720', u'test_collections-765', u'test_collections-760', u'test_collections-937', u'test_collections-737', u'test_collections-812', u'test_collections-854', u'test_collections-657', u'test_collections-811', u'test_collections-698', u'test_collections-829', u'test_collections-706', u'test_collections-905', u'test_collections-702', u'test_collections-502', u'test_collections-788', u'test_collections-940', u'test_collections-785', u'test_collections-860', u'test_collections-839', u'test_collections-715', u'test_collections-711', u'test_collections-676', u'test_collections-754', u'test_collections-795'] 2021-05-18 00:22:54,318 | test | WARNING | Thread-49015 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:POrYD-37-135000 update failed keys from task: [u'test_collections-724', u'test_collections-889', u'test_collections-922', u'test_collections-843', u'test_collections-888', u'test_collections-723', u'test_collections-720', u'test_collections-764', u'test_collections-765', u'test_collections-960', u'test_collections-884', u'test_collections-760', u'test_collections-937', u'test_collections-737', u'test_collections-859', u'test_collections-977', u'test_collections-854', u'test_collections-811', u'test_collections-932', u'test_collections-972', u'test_collections-772', u'test_collections-829', u'test_collections-706', u'test_collections-702', u'test_collections-703', u'test_collections-824', u'test_collections-788', u'test_collections-789', u'test_collections-743', u'test_collections-861', u'test_collections-785', u'test_collections-860', u'test_collections-839', u'test_collections-715', u'test_collections-795'] 2021-05-18 00:22:54,367 | test | WARNING | Thread-49018 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:yr8TfIij5m-37-136000 delete failed keys from task: [u'test_collections-248', u'test_collections-322', u'test_collections-484', u'test_collections-219', u'test_collections-337', u'test_collections-213', u'test_collections-299', u'test_collections-211', u'test_collections-252', u'test_collections-370', u'test_collections-293', u'test_collections-309', u'test_collections-428', u'test_collections-346', u'test_collections-303', u'test_collections-389', u'test_collections-220', u'test_collections-383', u'test_collections-381', u'test_collections-438', u'test_collections-479', u'test_collections-355', u'test_collections-235', u'test_collections-232', u'test_collections-431', u'test_collections-350', u'test_collections-270'] 2021-05-18 00:22:54,499 | test | WARNING | Thread-49033 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:pDj3K8QozHRUf2Q-37-136000 delete failed keys from task: [u'test_collections-450', u'test_collections-492'] 2021-05-18 00:22:54,513 | test | WARNING | Thread-48984 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 update failed keys from task: [u'test_collections-849', u'test_collections-806', u'test_collections-769', u'test_collections-887', u'test_collections-842', u'test_collections-520', u'test_collections-880', u'test_collections-937', u'test_collections-657', u'test_collections-698', u'test_collections-776', u'test_collections-773', u'test_collections-894', u'test_collections-707', u'test_collections-905', u'test_collections-869', u'test_collections-748', u'test_collections-705', u'test_collections-502', u'test_collections-821', u'test_collections-547', u'test_collections-745', u'test_collections-940', u'test_collections-864', u'test_collections-780', u'test_collections-676', u'test_collections-710'] 2021-05-18 00:22:54,644 | test | WARNING | Thread-49026 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:Om4-37-137000 update failed keys from task: [u'test_collections-519', u'test_collections-834', u'test_collections-602', u'test_collections-597', u'test_collections-598', u'test_collections-871', u'test_collections-682', u'test_collections-580'] 2021-05-18 00:22:54,673 | test | WARNING | Thread-49017 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:collection2 update failed keys from task: [u'test_collections-926', u'test_collections-729', u'test_collections-847', u'test_collections-969', u'test_collections-723', u'test_collections-720', u'test_collections-765', u'test_collections-938', u'test_collections-979', u'test_collections-812', u'test_collections-898', u'test_collections-854', u'test_collections-897', u'test_collections-771', u'test_collections-829', u'test_collections-906', u'test_collections-825', u'test_collections-902', u'test_collections-702', u'test_collections-788', u'test_collections-786', u'test_collections-860', u'test_collections-916', u'test_collections-837', u'test_collections-878', u'test_collections-711', u'test_collections-910', u'test_collections-951', u'test_collections-754', u'test_collections-872'] 2021-05-18 00:22:55,092 | test | WARNING | Thread-49031 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:oDYQlY0uYfeL1mS60-37-136000 update failed keys from task: [u'test_collections-519', u'test_collections-503', u'test_collections-558', u'test_collections-679', u'test_collections-544', u'test_collections-682', u'test_collections-580'] 2021-05-18 00:22:55,269 | test | WARNING | Thread-48976 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:BQ2LS-37-134000 delete failed keys from task: [u'test_collections-14', u'test_collections-15', u'test_collections-11', u'test_collections-99', u'test_collections-56', u'test_collections-13', u'test_collections-205', u'test_collections-51', u'test_collections-368', u'test_collections-485', u'test_collections-480', u'test_collections-46', u'test_collections-216', u'test_collections-177', u'test_collections-132', u'test_collections-253', u'test_collections-77', u'test_collections-109', u'test_collections-227', u'test_collections-425', u'test_collections-73', u'test_collections-228', u'test_collections-105', u'test_collections-223', u'test_collections-386', u'test_collections-266', u'test_collections-186', u'test_collections-187', u'test_collections-143', u'test_collections-460', u'test_collections-182', u'test_collections-0', u'test_collections-118', u'test_collections-64', u'test_collections-114', u'test_collections-395', u'test_collections-151', u'test_collections-190'] 2021-05-18 00:22:56,098 | test | WARNING | Thread-48923 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:MUK8t5s2-37-137000:7-37-139000 delete failed keys from task: [u'test_collections-18', u'test_collections-247', u'test_collections-324', u'test_collections-323', u'test_collections-202', u'test_collections-162', u'test_collections-240', u'test_collections-361', u'test_collections-480', u'test_collections-88', u'test_collections-138', u'test_collections-42', u'test_collections-215', u'test_collections-377', u'test_collections-210', u'test_collections-255', u'test_collections-250', u'test_collections-76', u'test_collections-35', u'test_collections-425', u'test_collections-74', u'test_collections-149', u'test_collections-101', u'test_collections-385', u'test_collections-460', u'test_collections-380', u'test_collections-61', u'test_collections-239', u'test_collections-115', u'test_collections-71', u'test_collections-194', u'test_collections-392', u'test_collections-191'] 2021-05-18 00:22:56,384 | test | WARNING | Thread-48951 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:fZH4weINYdYu2KP7UCR-37-140000:jOHz-37-142000 delete failed keys from task: [u'test_collections-18', u'test_collections-407', u'test_collections-324', u'test_collections-162', u'test_collections-361', u'test_collections-88', u'test_collections-138', u'test_collections-42', u'test_collections-215', u'test_collections-210', u'test_collections-255', u'test_collections-250', u'test_collections-76', u'test_collections-35', u'test_collections-74', u'test_collections-149', u'test_collections-101', u'test_collections-385', u'test_collections-380', u'test_collections-61', u'test_collections-239', u'test_collections-115', u'test_collections-71', u'test_collections-194', u'test_collections-191'] 2021-05-18 00:22:56,492 | test | WARNING | Thread-49012 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:qTdtMsCki0xtI%-37-136000 delete failed keys from task: [u'test_collections-58', u'test_collections-54', u'test_collections-405', u'test_collections-249', u'test_collections-125', u'test_collections-241', u'test_collections-414', u'test_collections-413', u'test_collections-373', u'test_collections-371', u'test_collections-370', u'test_collections-104', u'test_collections-346', u'test_collections-347', u'test_collections-221', u'test_collections-189', u'test_collections-220', u'test_collections-341', u'test_collections-67', u'test_collections-238', u'test_collections-478', u'test_collections-355', u'test_collections-207', u'test_collections-97', u'test_collections-204', u'test_collections-325', u'test_collections-321', u'test_collections-440', u'test_collections-280', u'test_collections-219', u'test_collections-84', u'test_collections-217', u'test_collections-214', u'test_collections-296', u'test_collections-451', u'test_collections-292', u'test_collections-293', u'test_collections-306', u'test_collections-428', u'test_collections-302', u'test_collections-303', u'test_collections-389', u'test_collections-387', u'test_collections-384', u'test_collections-397', u'test_collections-310', u'test_collections-394'] 2021-05-18 00:23:10,358 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 00:24:31,022 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 95% 2021-05-18 00:24:31,023 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 96% 2021-05-18 00:24:52,732 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 95% 2021-05-18 00:24:52,733 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 95% 2021-05-18 00:25:15,648 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 00:25:15,650 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 00:26:16,654 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.126999855 seconds to complete 2021-05-18 00:26:16,674 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2876.229s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 21 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_21 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_22 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_22', 'nodes_init': '5', 'GROUP': 'rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 22, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 00:26:26,256 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #22 test_data_load_collections_with_rebalance_out ========= 2021-05-18 00:26:26,951 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #22 test_data_load_collections_with_rebalance_out ========= 2021-05-18 00:26:27,384 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x62de12 type=client open_count=1 channel=[id: 0x739e0e9f, 0.0.0.0/0.0.0.0:34837] timeout=300.0>) 2021-05-18 00:26:48,155 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x62de13 type=client open_count=1 channel=[id: 0x84f84af1, 0.0.0.0/0.0.0.0:41589] timeout=300.0>) 2021-05-18 00:26:50,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x62de14 type=client open_count=1 channel=[id: 0xd22f3f33, 0.0.0.0/0.0.0.0:32933] timeout=300.0>) 2021-05-18 00:26:52,198 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x62de15 type=client open_count=1 channel=[id: 0xee3e1b53, 0.0.0.0/0.0.0.0:47380] timeout=300.0>) 2021-05-18 00:27:16,424 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x62de16 type=client open_count=1 channel=[id: 0xd8056dc6, 0.0.0.0/0.0.0.0:44062] timeout=300.0>) 2021-05-18 00:27:18,443 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x62de17 type=client open_count=1 channel=[id: 0x541df0e4, 0.0.0.0/0.0.0.0:45144] timeout=300.0>) 2021-05-18 00:27:20,460 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x62de18 type=client open_count=1 channel=[id: 0x7816d2cc, 0.0.0.0/0.0.0.0:36505] timeout=300.0>) 2021-05-18 00:27:44,721 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x62de19 type=client open_count=1 channel=[id: 0xc3efbfe9, 0.0.0.0/0.0.0.0:40143] timeout=300.0>) 2021-05-18 00:27:46,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x62de1a type=client open_count=1 channel=[id: 0x6e7d8960, 0.0.0.0/0.0.0.0:35362] timeout=300.0>) 2021-05-18 00:27:48,762 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x62de1b type=client open_count=1 channel=[id: 0x7cd5fd5d, 0.0.0.0/0.0.0.0:47707] timeout=300.0>) 2021-05-18 00:28:13,063 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x62de1c type=client open_count=1 channel=[id: 0x20b1be35, 0.0.0.0/0.0.0.0:36529] timeout=300.0>) 2021-05-18 00:28:15,082 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x62de1d type=client open_count=1 channel=[id: 0xb6cbe14c, 0.0.0.0/0.0.0.0:57015] timeout=300.0>) 2021-05-18 00:28:17,099 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x62de1e type=client open_count=1 channel=[id: 0x55b0229b, 0.0.0.0/0.0.0.0:37075] timeout=300.0>) 2021-05-18 00:28:41,773 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x62de1f type=client open_count=1 channel=[id: 0x2e02d6c9, 0.0.0.0/0.0.0.0:43225] timeout=300.0>) 2021-05-18 00:28:43,793 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x62de20 type=client open_count=1 channel=[id: 0x2872448d, 0.0.0.0/0.0.0.0:34006] timeout=300.0>) 2021-05-18 00:28:45,812 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 00:29:02,882 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 00:29:16,598 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 00:29:16,624 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #22 test_data_load_collections_with_rebalance_out ========= 2021-05-18 00:29:16,664 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #22 test_data_load_collections_with_rebalance_out ========= 2021-05-18 00:29:30,898 | test | INFO | pool-24-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.20483975801 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 00:29:30,924 | test | INFO | pool-24-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 00:29:41,005 | test | INFO | pool-24-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 00:29:41,030 | test | INFO | pool-24-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1319999695 sec 2021-05-18 00:29:41,308 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 00:30:26,842 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 00:30:27,153 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 708161432 | 85390061 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1048576000 | 752622272 | 98957707 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 1096039976 | 50791210 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-18 00:30:27,153 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 00:31:14,677 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 00:31:14,688 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 00:48:27,298 | test | INFO | pool-24-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................*****X | 52.485 | | 0.000 | ***********************..******...*****X | 2.996 | | 0.000 | *****..............******...****X | 8.992 | | 4.990 | ........X | 4.990 | | 1.498 | .****X | 1.498 | | 1.398 | .X | 1.398 | | 1.099 | .*****....****X | 3.397 | | 2.597 | ...X | 2.597 | | 0.800 | *****.*..****X | 2.197 | | 0.000 | *****************......*..****X | 4.497 | | 3.798 | ......X | 3.798 | | 2.998 | ....****X | 2.998 | | 1.499 | .****X | 1.499 | | 1.499 | .*****X | 1.499 | | 0.000 | *****..******...*****X | 2.998 | | 0.000 | ************.....****X | 2.998 | | 0.000 | *****************************........*****......*...................****X | 17.793 | | 5.998 | ..........****X | 5.998 | | 0.000 | ******.................****X | 8.998 | | 1.800 | ..X | 1.800 | | 0.000 | ****************..*****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | **********************..*****.......................*****X | 13.495 | | 1.500 | .*****.........******X | 6.000 | | 6.000 | ..........****X | 6.000 | | 5.998 | ..........**********X | 5.999 | | 0.000 | **********************************..******......****X | 4.500 | | 4.099 | ......******..*****......*.****X | 8.594 | | 2.401 | ...*****.......*..***X | 7.000 | | 6.200 | ..........X | 6.200 | | 5.298 | .........***X | 5.298 | | 2.299 | ...X | 2.299 | | 0.300 | ****X | 0.300 | | 0.000 | ***********************************...........*****X | 6.005 | | 4.504 | .......*****X | 4.504 | | 0.000 | *****.....*****..*.*****.*..****X | 6.003 | | 3.403 | .....*****.....*..****X | 7.102 | | 3.602 | .....X | 3.602 | | 0.000 | **********........*****.*.*.****X | 6.103 | | 6.102 | ..........*****...*..****X | 8.602 | | 5.203 | ........X | 5.203 | | 0.400 | ****X | 0.400 | | 0.000 | *****..*........****X | 5.502 | | 4.802 | ........X | 4.802 | | 1.997 | ..****X | 1.997 | | 0.000 | *****...*.....*...***X | 6.001 | | 5.401 | .........X | 5.401 | | 2.999 | ....*****...****X | 4.503 | | 1.502 | .X | 1.502 | | 0.000 | ********************************X | 0.000 | +-------+-------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 00:48:27,992 | test | INFO | pool-24-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ******..........................*..........................................................................****X | 47.987 | | 0.000 | ***********************......*****X | 2.996 | | 0.000 | ******...******............*****..*.......****X | 11.989 | | 4.096 | .......****X | 4.096 | | 2.296 | ...X | 2.296 | | 1.898 | ..*****......*****X | 4.496 | | 4.296 | .......X | 4.296 | | 3.697 | ......**********X | 3.796 | | 0.000 | *****************............***********X | 5.996 | | 1.499 | ..****X | 1.499 | | 1.499 | ..******......****X | 4.497 | | 2.999 | .....******............*****X | 8.995 | | 1.499 | ..****X | 1.499 | | 1.099 | .X | 1.099 | | 0.000 | ****************************************.*****...******..........****X | 7.098 | | 1.999 | ...******.....................****X | 11.997 | | 3.599 | ......X | 3.599 | | 1.500 | ..****X | 1.500 | | 0.000 | *************...*****...****X | 2.999 | | 1.500 | ..*****X | 1.500 | | 0.000 | ****************............*****X | 5.898 | | 1.600 | ..*****.........*****X | 5.997 | | 3.000 | .....*****.*.....****X | 5.998 | | 1.500 | ..******.*****.*****X | 2.500 | | 2.300 | ...*.****X | 2.699 | | 1.800 | ..****X | 1.800 | | 0.000 | *****************............*****X | 6.000 | | 0.200 | *****.................*......*.......***X | 14.404 | | 7.803 | ...............X | 7.803 | | 5.996 | ...........****X | 5.996 | | 3.101 | .....*****......*.***X | 6.500 | | 5.000 | .........X | 5.000 | | 0.900 | *****..****X | 1.503 | | 0.403 | X | 0.403 | | 0.000 | **********************************.....*****X | 2.703 | | 0.300 | ******......*****...................*****X | 12.008 | | 0.000 | ******......****X | 3.001 | | 1.402 | .X | 1.402 | | 0.000 | ****************.*..*****.........****X | 6.002 | | 3.903 | .......X | 3.903 | | 1.501 | ..*****......******................*****X | 12.303 | | 2.700 | ....*****X | 2.700 | | 0.000 | *****..................*****X | 8.902 | | 1.599 | ..****X | 1.599 | | 0.900 | X | 0.900 | | 0.000 | *****..............*........****X | 10.903 | | 6.702 | ............X | 6.702 | | 1.099 | .*****.****X | 1.501 | | 0.501 | X | 0.501 | | 0.000 | *********************************X | 0.000 | +-------+------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 00:48:28,979 | test | INFO | pool-24-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ******..............................................................................*....................*****..****X | 781.570 | | 650.774 | ..................................................................................******.*...****X | 684.130 | | 576.211 | ........................................................................******.......****X | 632.834 | | 602.451 | ............................................................................X | 602.451 | | 573.674 | ........................................................................*****X | 574.188 | | 555.245 | ......................................................................****X | 555.245 | | 447.725 | ........................................................****X | 447.725 | | 441.267 | .......................................................X | 441.267 | | 416.849 | ....................................................**********..******............******.****X | 535.427 | | 460.083 | .........................................................X | 460.083 | | 434.879 | ......................................................*****.....................*****X | 599.109 | | 511.645 | ................................................................X | 511.645 | | 473.393 | ...........................................................*****............*****X | 564.228 | | 534.778 | ...................................................................*****.....*****X | 571.321 | | 558.858 | ......................................................................****X | 558.858 | | 438.467 | .......................................................*****X | 438.467 | | 423.572 | .....................................................*****X | 423.572 | | 407.382 | ...................................................*****..*....*****...........*****X | 544.711 | | 538.440 | ...................................................................*****X | 538.440 | | 519.324 | .................................................................X | 519.324 | | 504.386 | ...............................................................***X | 504.386 | | 472.313 | ...........................................................X | 472.313 | | 452.447 | ........................................................*****.........*****X | 520.356 | | 496.918 | ..............................................................*****...........*...****X | 609.586 | | 564.183 | .......................................................................X | 564.183 | | 505.291 | ...............................................................****X | 505.291 | | 493.792 | ..............................................................****X | 493.792 | | 422.961 | .....................................................******..*****X | 442.278 | | 439.496 | .......................................................X | 439.496 | | 435.971 | ......................................................****X | 435.971 | | 434.496 | ......................................................*****...........*****X | 520.231 | | 448.581 | ........................................................******........*****..**********X | 526.717 | | 451.938 | ........................................................X | 451.938 | | 436.864 | ......................................................*****..........****X | 511.258 | | 446.720 | ........................................................******.......****X | 505.974 | | 478.576 | ............................................................X | 478.576 | | 461.649 | ..........................................................*****.........****X | 532.567 | | 429.922 | ......................................................***********X | 437.597 | | 395.801 | .................................................****X | 395.801 | | 394.102 | .................................................*..........****X | 469.225 | | 445.623 | ........................................................X | 445.623 | | 424.498 | .....................................................*****.........................*****X | 622.727 | | 562.700 | ......................................................................*****X | 562.700 | | 424.748 | .....................................................****X | 424.748 | | 399.672 | ..................................................*****...*....*****.......*****.X | 516.102 | | 504.626 | ...............................................................*****..*****X | 516.909 | | 419.166 | ....................................................X | 419.166 | | 386.476 | ................................................****...*..****X | 423.472 | | 408.101 | ...................................................****X | 408.101 | | 401.768 | ..................................................*.****X | 409.885 | | 391.659 | .................................................****.........*.....****X | 504.230 | | 463.739 | ..........................................................****X | 467.953 | | 451.154 | ........................................................X | 451.154 | | 425.580 | .....................................................*****..******.***********..*...****X | 488.730 | | 382.543 | ...............................................*****X | 382.543 | | 326.177 | ........................................******................****X | 449.303 | | 425.825 | .....................................................*****.*..****X | 446.836 | | 414.447 | ....................................................******.....****X | 459.945 | | 453.770 | .........................................................X | 453.770 | | 448.086 | ........................................................*****.........*****X | 519.341 | | 428.567 | .....................................................*****...*****.*.......**********X | 513.541 | | 462.347 | ..........................................................****X | 462.347 | | 437.892 | .......................................................X | 437.892 | | 437.847 | .......................................................*****.........*****X | 513.321 | | 449.963 | ........................................................X | 449.963 | | 426.425 | .....................................................****..*....****X | 475.839 | | 458.598 | .........................................................****X | 458.598 | | 443.679 | .......................................................X | 443.679 | | 381.464 | ...............................................*****.*.....*****...*****..******.......****X | 518.344 | | 495.398 | ..............................................................X | 495.398 | | 478.234 | ............................................................X | 478.234 | | 468.001 | ..........................................................****.......*...*****..........*****X | 618.707 | | 572.653 | ........................................................................*****X | 572.653 | | 282.094 | ...................................X | 282.094 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 00:48:29,286 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 75.7812630211 | 6437847040 | 4281593856 | 2433970176 / 7511588864 | 101978 / 205348 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 58.8026467108 | 6437847040 | 4197543936 | 2534879232 / 7511588864 | 99389 / 202076 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 80.0959673644 | 6437847040 | 4263546880 | 2465783808 / 7511588864 | 100808 / 201384 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 66.0416666667 | 6437847040 | 3886944256 | 2951884800 / 9530753024 | 100982 / 199759 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 67.2377872965 | 6437847040 | 4240941056 | 2508410880 / 7511588864 | 100984 / 199782 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 00:48:41,996 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 00:50:10,155 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 164377224 | 250428086 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162215968 | 264279875 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 442926440 | 463959980 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-18 00:50:10,157 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #22 test_data_load_collections_with_rebalance_out ========= 2021-05-18 00:50:20,193 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_out 2021-05-18 00:50:20,194 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-18 00:50:20,263 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 00:50:20,513 | test | INFO | pool-24-thread-20 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 7.86653555774 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 8.1513949535 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.61301289957 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 16.6138871296 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 12.0566666667 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 00:50:20,549 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 00:50:28,993 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 00:50:29,500 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 00:50:34,654 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 1.73 2021-05-18 00:50:39,786 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 3.69 2021-05-18 00:50:44,915 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 5.58 2021-05-18 00:50:50,006 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 8.09 2021-05-18 00:50:55,148 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 10.46 2021-05-18 00:50:59,186 | test | INFO | pool-24-thread-30 | [table_view:display:72] Keys failed in bucket1:scope1:PF3AvWJTbS_yjSCj46K5-41-378000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:51:00,273 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.02 2021-05-18 00:51:05,368 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 13.49 2021-05-18 00:51:10,470 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 15.15 2021-05-18 00:51:14,423 | test | INFO | pool-24-thread-4 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:5hhWRLKEzU_DI7fIzXIP-26-925000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:51:14,424 | test | INFO | pool-24-thread-4 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:5hhWRLKEzU_DI7fIzXIP-26-925000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-254 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:51:15,562 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 16.41 2021-05-18 00:51:20,670 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 17.57 2021-05-18 00:51:25,766 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 18.99 2021-05-18 00:51:30,898 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 20.05 2021-05-18 00:51:36,000 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 21.37 2021-05-18 00:51:37,862 | test | INFO | pool-24-thread-28 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:mji1CQx5YdF4StXuHk-26-927000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-532 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:51:37,864 | test | INFO | pool-24-thread-28 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:mji1CQx5YdF4StXuHk-26-927000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-532 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:51:41,121 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 22.8 2021-05-18 00:51:46,246 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 24.32 2021-05-18 00:51:51,331 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 25.52 2021-05-18 00:51:56,438 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 26.85 2021-05-18 00:52:01,561 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 28.15 2021-05-18 00:52:06,634 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 29.67 2021-05-18 00:52:08,460 | test | INFO | pool-24-thread-28 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:mji1CQx5YdF4StXuHk-26-927000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:08,460 | test | INFO | pool-24-thread-28 | [table_view:display:72] Keys failed in bucket1:c1ayG-26-758000:mji1CQx5YdF4StXuHk-26-927000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:11,723 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 31.16 2021-05-18 00:52:16,502 | test | INFO | pool-24-thread-10 | [table_view:display:72] Keys failed in bucket1:dGYBf-26-252000:BDcfvSyOcLz3FM0hPBXr-26-425000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:16,503 | test | INFO | pool-24-thread-14 | [table_view:display:72] Keys failed in bucket1:dGYBf-26-252000:EVPNzcZRkQk-26-425000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:16,503 | test | INFO | pool-24-thread-10 | [table_view:display:72] Keys failed in bucket1:dGYBf-26-252000:BDcfvSyOcLz3FM0hPBXr-26-425000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:16,503 | test | INFO | pool-24-thread-14 | [table_view:display:72] Keys failed in bucket1:dGYBf-26-252000:EVPNzcZRkQk-26-425000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:16,793 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-18 00:52:21,871 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 00:52:26,961 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 00:52:32,131 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 00:52:37,259 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 35.05 2021-05-18 00:52:42,401 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 38.22 2021-05-18 00:52:47,515 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 41.06 2021-05-18 00:52:52,628 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 42.84 2021-05-18 00:52:57,743 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 44.29 2021-05-18 00:52:59,582 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:00%2d-24-979000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3002 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:52:59,584 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:00%2d-24-979000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3192 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3002 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:53:00,469 | test | INFO | pool-24-thread-26 | [table_view:display:72] Keys failed in bucket2:_default:CPqP6hPwnPk7RK56wCKo-41-357000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:53:00,568 | test | INFO | pool-24-thread-29 | [table_view:display:72] Keys failed in bucket2:eWYYLoHRX78w2dvN9NR-41-369000:7exl-41-373000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:53:02,823 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 45.96 2021-05-18 00:53:07,936 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 47.6 2021-05-18 00:53:08,977 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:00%2d-24-979000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3240 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:53:08,977 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:00%2d-24-979000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3240 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:53:13,039 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 49.07 2021-05-18 00:53:18,173 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 50.73 2021-05-18 00:53:23,299 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 52.22 2021-05-18 00:53:28,417 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 53.41 2021-05-18 00:53:33,530 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 55.1 2021-05-18 00:53:38,663 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 56.63 2021-05-18 00:53:43,759 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 58.06 2021-05-18 00:53:48,839 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 59.46 2021-05-18 00:53:53,926 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 61.0 2021-05-18 00:53:59,006 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 62.5 2021-05-18 00:54:04,095 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 64.03 2021-05-18 00:54:09,191 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 65.59 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in bucket2:hkQMCv9-25-801000:VzhFG-26-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:4NZqvKlLNit-24-978000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-19 | [table_view:display:72] Keys failed in bucket2:kCzbY_WPfr7CWN9D-25-259000:kl_-25-461000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-16 | [table_view:display:72] Keys failed in bucket2:hkQMCv9-25-801000:8_M2IB20swUiE%%-26-9000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-2 | [table_view:display:72] Keys failed in bucket2:kCzbY_WPfr7CWN9D-25-259000:Zwwui4zxWTvJvRVb-25-462000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,051 | test | INFO | pool-24-thread-16 | [table_view:display:72] Keys failed in bucket2:hkQMCv9-25-801000:8_M2IB20swUiE%%-26-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in bucket2:hkQMCv9-25-801000:VzhFG-26-9000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,049 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in bucket2:Zb-24-863000:4NZqvKlLNit-24-978000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,051 | test | INFO | pool-24-thread-19 | [table_view:display:72] Keys failed in bucket2:kCzbY_WPfr7CWN9D-25-259000:kl_-25-461000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:11,051 | test | INFO | pool-24-thread-2 | [table_view:display:72] Keys failed in bucket2:kCzbY_WPfr7CWN9D-25-259000:Zwwui4zxWTvJvRVb-25-462000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4361 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:14,269 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 00:54:19,362 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 00:54:26,203 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 00:54:31,328 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 69.66 2021-05-18 00:54:35,990 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:scope1:jrCK-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-874 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:36,440 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 71.63 2021-05-18 00:54:41,707 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 73.96 2021-05-18 00:54:41,736 | test | INFO | pool-24-thread-24 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:EpqKc3M7N1nrg-41-403000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:46,795 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 76.02 2021-05-18 00:54:49,460 | test | INFO | pool-24-thread-18 | [table_view:display:72] Keys failed in default:_default:aht-41-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-749 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-752 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:50,236 | test | INFO | pool-24-thread-26 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:51,914 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 77.55 2021-05-18 00:54:57,013 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 79.37 2021-05-18 00:54:58,155 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:scope1:jrCK-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-919 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-938 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:59,178 | test | INFO | pool-24-thread-12 | [table_view:display:72] Keys failed in default:scope1:Cwq0g22-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-539 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-500 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-545 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:54:59,697 | test | INFO | pool-24-thread-27 | [table_view:display:72] Keys failed in default:_default:tHKP_jvgHxIDo4i-41-390000 +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-66 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-30 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +---------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:02,180 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 80.75 2021-05-18 00:55:02,746 | test | INFO | pool-24-thread-18 | [table_view:display:72] Keys failed in default:_default:aht-41-391000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-929 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-903 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:04,842 | test | INFO | pool-24-thread-26 | [table_view:display:72] Keys failed in default:_default:_default +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-910 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-950 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:07,263 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 81.94 2021-05-18 00:55:12,352 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 83.63 2021-05-18 00:55:14,861 | test | INFO | pool-24-thread-27 | [table_view:display:72] Keys failed in default:_default:tHKP_jvgHxIDo4i-41-390000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-229 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-368 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-314 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-311 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-232 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-320 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-354 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-343 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-372 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-284 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-281 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:14,924 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in default:scope1:470caW2rUu6lbLUh-41-392000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-590 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:14,924 | test | INFO | pool-24-thread-29 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:8s0-41-404000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-641 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:14,924 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:FZh4ROaTt7-41-402000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-590 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:14,926 | test | INFO | pool-24-thread-3 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:rAMIjetKObIlymIU6O-41-402000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-669 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-540 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-673 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-660 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-590 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:17,437 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 85.52 2021-05-18 00:55:22,523 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 87.42 2021-05-18 00:55:22,576 | test | INFO | pool-24-thread-12 | [table_view:display:72] Keys failed in default:scope1:Cwq0g22-41-393000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-819 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-810 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-777 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-842 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:22,578 | test | INFO | pool-24-thread-24 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:EpqKc3M7N1nrg-41-403000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-819 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-838 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-706 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-777 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-842 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-743 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-820 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-851 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:27,637 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 89.28 2021-05-18 00:55:29,977 | test | INFO | pool-24-thread-13 | [table_view:display:72] Keys failed in default:o8Pl3a3OZ9-q%sEiWLN-27-997000:NyTHeAArkL1T7Jbc9-28-144000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:29,979 | test | INFO | pool-24-thread-13 | [table_view:display:72] Keys failed in default:o8Pl3a3OZ9-q%sEiWLN-27-997000:NyTHeAArkL1T7Jbc9-28-144000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4132 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:30,607 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-37 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:30,671 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-628 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-679 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-577 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-543 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-552 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-681 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-580 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:32,720 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 91.14 2021-05-18 00:55:33,798 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:FZh4ROaTt7-41-402000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:33,798 | test | INFO | pool-24-thread-3 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:rAMIjetKObIlymIU6O-41-402000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:33,798 | test | INFO | pool-24-thread-25 | [table_view:display:72] Keys failed in default:scope1:470caW2rUu6lbLUh-41-392000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:33,798 | test | INFO | pool-24-thread-29 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:8s0-41-404000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-837 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-711 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-797 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-754 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:37,828 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 92.86 2021-05-18 00:55:38,440 | test | INFO | pool-24-thread-2 | [table_view:display:72] Keys failed in default:scope1:lJy6EbB-d46P-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-37 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-175 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:38,832 | test | INFO | pool-24-thread-24 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:EpqKc3M7N1nrg-41-403000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:38,832 | test | INFO | pool-24-thread-12 | [table_view:display:72] Keys failed in default:scope1:Cwq0g22-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-958 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-922 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-953 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-921 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:42,921 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 94.59 2021-05-18 00:55:47,575 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-309 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-334 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-268 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-365 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-299 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-274 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-361 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:47,683 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-807 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-718 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-803 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-868 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-813 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-744 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:48,068 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 96.21 2021-05-18 00:55:53,144 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 97.69 2021-05-18 00:55:53,644 | test | INFO | pool-24-thread-30 | [table_view:display:72] Keys failed in default:o8Pl3a3OZ9-q%sEiWLN-27-997000:o34oh0CbPwbzU0A30-28-143000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4135 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:53,645 | test | INFO | pool-24-thread-30 | [table_view:display:72] Keys failed in default:o8Pl3a3OZ9-q%sEiWLN-27-997000:o34oh0CbPwbzU0A30-28-143000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4131 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4135 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:53,648 | test | INFO | pool-24-thread-17 | [table_view:display:72] Keys failed in default:E7ywbVj4zg2CkO-Z-28-369000:pfB--28-515000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4135 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:53,650 | test | INFO | pool-24-thread-17 | [table_view:display:72] Keys failed in default:E7ywbVj4zg2CkO-Z-28-369000:pfB--28-515000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4131 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4135 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:54,246 | test | INFO | pool-24-thread-7 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:FZh4ROaTt7-41-402000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-924 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:54,250 | test | INFO | pool-24-thread-29 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:8s0-41-404000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-916 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-924 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-977 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:55:58,259 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 99.03 2021-05-18 00:56:03,345 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 00:56:03,483 | test | INFO | pool-24-thread-3 | [table_view:display:72] Keys failed in default:wGqaXE6Y-41-401000:rAMIjetKObIlymIU6O-41-402000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-924 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-945 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-932 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-971 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-961 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-970 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:04,058 | test | INFO | pool-24-thread-19 | [table_view:display:72] Keys failed in default:scope1:SXSbEBHjeebo-41-395000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-47 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-12 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-79 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-35 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-139 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-70 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-111 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-82 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-142 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-154 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-130 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-160 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-181 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:04,105 | test | INFO | pool-24-thread-9 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-927 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-900 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-962 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:04,380 | test | INFO | pool-24-thread-2 | [table_view:display:72] Keys failed in default:scope1:lJy6EbB-d46P-41-393000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-319 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-227 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-203 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-278 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-289 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-213 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-393 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-383 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-350 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:07,335 | test | INFO | pool-24-thread-8 | [table_view:display:72] Keys failed in default:scope1:GMyjIa1C8-41-395000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-419 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-467 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-468 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-455 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-480 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:08,417 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 00:56:10,447 | test | INFO | pool-24-thread-5 | [table_view:display:72] Keys failed in default:7%OQUObXkZ-27-593000:uvvQ_vf83lS6-27-743000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:10,447 | test | INFO | pool-24-thread-5 | [table_view:display:72] Keys failed in default:7%OQUObXkZ-27-593000:uvvQ_vf83lS6-27-743000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2918 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2888 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 00:56:13,971 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 00:56:24,216 | test | INFO | pool-24-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 00:56:24,401 | test | INFO | pool-24-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 363.888999939 sec 2021-05-18 00:56:50,078 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,079 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,079 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,081 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,081 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,082 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,082 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:50,082 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-18 00:56:50,084 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 00:56:50,084 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 00:56:53,598 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 00:56:56,318 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 00:56:56,323 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:56:56,325 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 00:58:59,970 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 00:58:59,974 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 00:58:59,976 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 00:58:59,980 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 4 docs from 500 to 980 2021-05-18 00:58:59,982 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-18 00:58:59,983 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 3 docs from 500 to 980 2021-05-18 00:58:59,983 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-18 00:58:59,983 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 13 docs from 0 to 500 2021-05-18 00:58:59,984 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-18 00:58:59,986 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 10 docs from 500 to 980 2021-05-18 00:58:59,986 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-18 00:58:59,986 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 18 docs from 500 to 980 2021-05-18 00:58:59,986 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-18 00:58:59,987 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 13 docs from 0 to 500 2021-05-18 00:58:59,989 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 19 docs from 500 to 980 2021-05-18 00:58:59,989 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 18 docs from 0 to 500 2021-05-18 00:58:59,989 | test | ERROR | pool-24-thread-15 | [task:call:3809] Failed to load 16 docs from 0 to 500 2021-05-18 01:03:28,947 | test | INFO | pool-24-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..*****...........................****......*..................****............................***X | 170.329 | | 141.526 | ..................................................................*****.......*.....***X | 166.125 | | 124.064 | ..........................................................****....X | 134.138 | | 133.347 | ..............................................................****...........****...............*......**X | 199.823 | | 166.760 | ..............................................................................X | 166.760 | | 152.941 | .......................................................................****............................***X | 209.898 | | 192.510 | ..........................................................................................***X | 192.510 | | 104.105 | ................................................*****...........***X | 128.011 | | 97.910 | .............................................X | 97.910 | | 96.902 | .............................................****........***X | 114.689 | | 91.198 | ..........................................X | 91.198 | | 50.358 | ......................****..............**X | 79.240 | | 79.240 | ....................................*X | 79.240 | | 58.293 | ..........................****.....*****...****X | 73.904 | | 71.241 | ................................*****.......................................****X | 152.323 | | 79.832 | .....................................****................***X | 113.545 | | 21.106 | .........****X | 21.106 | | 3.398 | ****X | 3.398 | | 0.500 | ***X | 0.500 | | 0.000 | ********X | 1.000 | | 0.200 | ****X | 0.200 | | 0.000 | **********.*****X | 2.800 | | 1.100 | ***********X | 1.400 | | 1.000 | X | 1.000 | | 0.600 | *****..*****X | 4.600 | | 3.199 | *****X | 3.199 | | 0.800 | *****X | 0.800 | | 0.599 | ************.X | 2.899 | | 2.899 | **********X | 2.900 | | 0.900 | X | 0.900 | | 0.500 | ************.******.*****.******X | 7.398 | | 2.300 | ****************X | 3.400 | | 2.800 | *X | 3.299 | | 3.299 | ***X | 3.299 | | 0.200 | ******...*****X | 6.300 | | 0.500 | ******.*.*****X | 5.900 | | 3.900 | **X | 3.900 | | 3.900 | **X | 3.900 | | 2.400 | X | 2.400 | | 2.000 | ****X | 2.000 | | 0.600 | *****X | 0.600 | | 0.400 | *******..****X | 4.299 | | 2.100 | *******.*****X | 4.900 | | 1.500 | ******..*****X | 5.100 | | 1.300 | ************....*****X | 8.801 | | 2.901 | *****X | 2.901 | | 1.900 | *******.*****.*****X | 5.201 | | 5.200 | .*****X | 5.800 | | 5.000 | .*.****X | 6.400 | | 0.000 | *******X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:03:29,381 | test | INFO | pool-24-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.****....****....*..***....*..........***X | 35.282 | | 14.505 | .........*****.........*.....****.....***X | 40.086 | | 30.907 | .....................X | 30.907 | | 23.206 | ................****.....................***X | 52.508 | | 43.507 | ..............................X | 43.507 | | 40.003 | ............................***........*...**X | 55.803 | | 48.511 | ..................................X | 48.511 | | 46.597 | .................................****..............................***X | 88.250 | | 73.803 | .....................................................****X | 73.803 | | 47.003 | .................................****........X | 57.405 | | 54.500 | ......................................****.....****....X | 65.794 | | 53.694 | ......................................X | 53.694 | | 48.755 | ..................................****........................****X | 81.348 | | 54.897 | .......................................****...............................****X | 97.619 | | 65.703 | ...............................................****..*....*****.........****...X | 91.277 | | 81.524 | ..........................................................***X | 81.524 | | 48.176 | ..................................****.....................****.....*........***X | 95.192 | | 85.669 | .............................................................*X | 85.669 | | 85.669 | .............................................................**...........*.....****..***X | 109.393 | | 98.098 | ......................................................................****X | 98.098 | | 98.037 | ......................................................................X | 98.037 | | 95.794 | .....................................................................****.............****X | 113.514 | | 106.293 | ............................................................................******.......*****X | 115.285 | | 105.304 | ............................................................................*****X | 105.304 | | 74.295 | .....................................................*****X | 74.295 | | 33.698 | .......................*****X | 33.698 | | 4.299 | ..*****...................*...........................................****X | 88.797 | | 82.476 | ...........................................................X | 82.476 | | 75.777 | ......................................................******..........******.................******.*...*****.******..*****X | 122.307 | | 115.704 | ...................................................................................X | 115.704 | | 109.296 | ...............................................................................*****......*..............****X | 136.486 | | 80.194 | .........................................................X | 80.194 | | 50.396 | ...................................*****X | 50.396 | | 1.800 | ************..****X | 4.400 | | 3.998 | .X | 3.998 | | 3.998 | .****X | 3.998 | | 0.901 | ******....**X | 5.600 | | 5.600 | ...**X | 5.600 | | 1.000 | ******..*.**********X | 5.300 | | 1.900 | *****X | 1.900 | | 1.300 | X | 1.300 | | 0.500 | ************..*******.****X | 4.601 | | 3.500 | .X | 3.500 | | 3.300 | .*****X | 3.300 | | 0.300 | ******....*****X | 6.000 | | 0.600 | ******.******..*....****X | 9.601 | | 4.801 | ..*****X | 4.801 | | 3.799 | .******.***********X | 5.400 | | 4.600 | ..*****...******X | 8.700 | | 0.000 | *******X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:03:30,513 | test | INFO | pool-24-thread-1 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.......................*......***X | 196.008 | | 87.415 | ............****..........*....***......................................*....................***X | 573.459 | | 218.943 | ...............................****..........*.****.........................................................***X | 669.218 | | 566.644 | ...................................................................................X | 566.644 | | 212.069 | ..............................****...*.........***X | 289.208 | | 234.828 | ..................................X | 234.828 | | 220.115 | ...............................***.....****X | 248.895 | | 229.689 | .................................****......................................***X | 485.782 | | 390.824 | .........................................................****X | 390.824 | | 236.919 | ..................................***X | 236.919 | | 222.495 | ................................X | 222.495 | | 179.404 | .........................***X | 179.404 | | 151.786 | .....................X | 151.786 | | 138.186 | ...................****........*.....****.....****X | 260.319 | | 140.989 | ....................****.......................................***X | 408.097 | | 283.990 | .........................................X | 283.990 | | 209.308 | ..............................***X | 209.308 | | 177.876 | .........................X | 177.876 | | 169.269 | ........................*****.......................................................***X | 541.447 | | 163.734 | .......................****.....***X | 195.048 | | 98.323 | .............****.......*............***X | 221.188 | | 109.465 | ...............****...........***X | 185.930 | | 151.879 | .....................X | 151.879 | | 105.392 | ..............****.*****.****.*****..........*..................*****X | 311.878 | | 260.467 | .....................................*X | 260.467 | | 260.467 | .....................................***X | 260.467 | | 215.209 | ...............................******..............................*X | 420.672 | | 420.672 | .............................................................***X | 420.672 | | 359.488 | ....................................................******..............................****X | 558.025 | | 432.582 | ...............................................................X | 432.582 | | 183.894 | ..........................****X | 183.894 | | 182.137 | ..........................X | 182.137 | | 175.039 | .........................*****.........*....................*****X | 373.788 | | 228.776 | .................................*****X | 228.776 | | 193.399 | ...........................******........******.....*****X | 276.618 | | 216.996 | ...............................******............******...........******..******........*****.X | 446.610 | | 426.040 | ..............................................................****X | 426.040 | | 376.777 | .......................................................******...............***X | 475.619 | | 475.619 | ......................................................................X | 475.619 | | 452.884 | ..................................................................X | 452.884 | | 378.287 | .......................................................******.............*****X | 464.415 | | 337.877 | .................................................*******..****X | 351.165 | | 275.992 | ........................................******..............*****X | 370.163 | | 368.963 | ......................................................*.****X | 380.461 | | 373.324 | ......................................................*.****X | 381.324 | | 378.186 | .......................................................X | 378.186 | | 360.286 | ....................................................*****X | 360.286 | | 313.189 | .............................................******..............*****X | 403.128 | | 380.877 | .......................................................X | 380.877 | | 354.700 | ....................................................****X | 354.700 | | 343.488 | ..................................................X | 343.488 | | 298.390 | ...........................................*****....*.................*****X | 435.213 | | 247.016 | ...................................******................******...*.*****......*****..*......****X | 469.348 | | 434.519 | ...............................................................X | 434.519 | | 407.273 | ...........................................................*****....*......*****.....*****X | 503.267 | | 392.467 | .........................................................X | 392.467 | | 192.394 | ...........................*X | 192.394 | +---------+------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:03:30,563 | test | WARNING | Thread-51445 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:CPqP6hPwnPk7RK56wCKo-41-357000 update failed keys from task: [u'test_collections-4'] 2021-05-18 01:03:30,566 | test | WARNING | Thread-51464 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:hkQMCv9-25-801000:VzhFG-26-9000 create failed keys from task: [u'test_collections-4361'] 2021-05-18 01:03:30,568 | test | WARNING | Thread-51465 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:hkQMCv9-25-801000:8_M2IB20swUiE%%-26-9000 create failed keys from task: [u'test_collections-4361'] 2021-05-18 01:03:30,628 | test | WARNING | Thread-51652 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:c1ayG-26-758000:5hhWRLKEzU_DI7fIzXIP-26-925000 create failed keys from task: [u'test_collections-254'] 2021-05-18 01:03:30,630 | test | WARNING | Thread-51655 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dGYBf-26-252000:BDcfvSyOcLz3FM0hPBXr-26-425000 create failed keys from task: [u'test_collections-335'] 2021-05-18 01:03:30,630 | test | WARNING | Thread-51659 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:PF3AvWJTbS_yjSCj46K5-41-378000 update failed keys from task: [u'test_collections-4'] 2021-05-18 01:03:30,733 | test | WARNING | Thread-51651 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:c1ayG-26-758000:mji1CQx5YdF4StXuHk-26-927000 create failed keys from task: [u'test_collections-978', u'test_collections-532'] 2021-05-18 01:03:30,630 | test | WARNING | Thread-51657 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:dGYBf-26-252000:EVPNzcZRkQk-26-425000 create failed keys from task: [u'test_collections-335'] 2021-05-18 01:03:30,753 | test | WARNING | Thread-51824 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:aht-41-391000 update failed keys from task: [u'test_collections-929', u'test_collections-749', u'test_collections-903', u'test_collections-955', u'test_collections-822', u'test_collections-910', u'test_collections-752'] 2021-05-18 01:03:30,755 | test | WARNING | Thread-51832 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:tHKP_jvgHxIDo4i-41-390000 delete failed keys from task: [u'test_collections-229', u'test_collections-30', u'test_collections-368', u'test_collections-320', u'test_collections-343', u'test_collections-284', u'test_collections-281', u'test_collections-66', u'test_collections-314', u'test_collections-311', u'test_collections-232', u'test_collections-354', u'test_collections-372'] 2021-05-18 01:03:30,756 | test | WARNING | Thread-51838 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:o8Pl3a3OZ9-q%sEiWLN-27-997000:NyTHeAArkL1T7Jbc9-28-144000 create failed keys from task: [u'test_collections-4132'] 2021-05-18 01:03:30,756 | test | WARNING | Thread-51840 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:E7ywbVj4zg2CkO-Z-28-369000:pfB--28-515000 create failed keys from task: [u'test_collections-4131', u'test_collections-4135'] 2021-05-18 01:03:30,757 | test | WARNING | Thread-51839 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:o8Pl3a3OZ9-q%sEiWLN-27-997000:o34oh0CbPwbzU0A30-28-143000 create failed keys from task: [u'test_collections-4131', u'test_collections-4135'] 2021-05-18 01:03:30,759 | test | WARNING | Thread-51849 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:GMyjIa1C8-41-395000 update failed keys from task: [u'test_collections-807', u'test_collections-607', u'test_collections-927', u'test_collections-803', u'test_collections-628', u'test_collections-548', u'test_collections-900', u'test_collections-868', u'test_collections-744', u'test_collections-962', u'test_collections-543', u'test_collections-681', u'test_collections-580', u'test_collections-718', u'test_collections-813', u'test_collections-679', u'test_collections-577', u'test_collections-552', u'test_collections-574'] 2021-05-18 01:03:30,829 | test | WARNING | Thread-51823 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:_default update failed keys from task: [u'test_collections-822', u'test_collections-910', u'test_collections-950'] 2021-05-18 01:03:30,835 | test | WARNING | Thread-51920 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:7%OQUObXkZ-27-593000:uvvQ_vf83lS6-27-743000 create failed keys from task: [u'test_collections-2918', u'test_collections-2888'] 2021-05-18 01:03:30,841 | test | WARNING | Thread-51567 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:eWYYLoHRX78w2dvN9NR-41-369000:7exl-41-373000 update failed keys from task: [u'test_collections-4'] 2021-05-18 01:03:30,867 | test | WARNING | Thread-51867 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:SXSbEBHjeebo-41-395000 delete failed keys from task: [u'test_collections-12', u'test_collections-79', u'test_collections-35', u'test_collections-106', u'test_collections-82', u'test_collections-142', u'test_collections-160', u'test_collections-181', u'test_collections-47', u'test_collections-69', u'test_collections-139', u'test_collections-135', u'test_collections-70', u'test_collections-111', u'test_collections-154', u'test_collections-130'] 2021-05-18 01:03:30,865 | test | WARNING | Thread-51861 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:470caW2rUu6lbLUh-41-392000 update failed keys from task: [u'test_collections-837', u'test_collections-669', u'test_collections-711', u'test_collections-599', u'test_collections-797', u'test_collections-754', u'test_collections-540', u'test_collections-673', u'test_collections-660', u'test_collections-590'] 2021-05-18 01:03:30,878 | test | WARNING | Thread-51614 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Zb-24-863000:4NZqvKlLNit-24-978000 create failed keys from task: [u'test_collections-4361'] 2021-05-18 01:03:30,878 | test | WARNING | Thread-51893 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:wGqaXE6Y-41-401000:EpqKc3M7N1nrg-41-403000 update failed keys from task: [u'test_collections-706', u'test_collections-922', u'test_collections-821', u'test_collections-921', u'test_collections-885', u'test_collections-842', u'test_collections-743', u'test_collections-820', u'test_collections-819', u'test_collections-838', u'test_collections-958', u'test_collections-777', u'test_collections-953', u'test_collections-773', u'test_collections-872', u'test_collections-851', u'test_collections-794', u'test_collections-593'] 2021-05-18 01:03:30,880 | test | WARNING | Thread-51868 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:jrCK-41-393000 update failed keys from task: [u'test_collections-919', u'test_collections-938', u'test_collections-955', u'test_collections-874'] 2021-05-18 01:03:30,881 | test | WARNING | Thread-51872 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:Cwq0g22-41-393000 update failed keys from task: [u'test_collections-706', u'test_collections-922', u'test_collections-821', u'test_collections-921', u'test_collections-500', u'test_collections-885', u'test_collections-545', u'test_collections-842', u'test_collections-743', u'test_collections-819', u'test_collections-539', u'test_collections-958', u'test_collections-810', u'test_collections-777', u'test_collections-953', u'test_collections-633', u'test_collections-773', u'test_collections-872'] 2021-05-18 01:03:30,918 | test | WARNING | Thread-51875 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:lJy6EbB-d46P-41-393000 delete failed keys from task: [u'test_collections-37', u'test_collections-227', u'test_collections-203', u'test_collections-289', u'test_collections-383', u'test_collections-25', u'test_collections-319', u'test_collections-278', u'test_collections-213', u'test_collections-175', u'test_collections-393', u'test_collections-350', u'test_collections-192'] 2021-05-18 01:03:30,921 | test | WARNING | Thread-51615 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:Zb-24-863000:00%2d-24-979000 create failed keys from task: [u'test_collections-3192', u'test_collections-3240', u'test_collections-3002'] 2021-05-18 01:03:30,924 | test | WARNING | Thread-51540 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kCzbY_WPfr7CWN9D-25-259000:Zwwui4zxWTvJvRVb-25-462000 create failed keys from task: [u'test_collections-4361'] 2021-05-18 01:03:30,924 | test | WARNING | Thread-51539 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:kCzbY_WPfr7CWN9D-25-259000:kl_-25-461000 create failed keys from task: [u'test_collections-4361'] 2021-05-18 01:03:30,928 | test | WARNING | Thread-51887 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:wGqaXE6Y-41-401000:rAMIjetKObIlymIU6O-41-402000 update failed keys from task: [u'test_collections-924', u'test_collections-669', u'test_collections-945', u'test_collections-961', u'test_collections-540', u'test_collections-660', u'test_collections-837', u'test_collections-978', u'test_collections-934', u'test_collections-711', u'test_collections-932', u'test_collections-599', u'test_collections-797', u'test_collections-754', u'test_collections-971', u'test_collections-673', u'test_collections-970', u'test_collections-590'] 2021-05-18 01:03:30,963 | test | WARNING | Thread-51899 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:wGqaXE6Y-41-401000:8s0-41-404000 update failed keys from task: [u'test_collections-924', u'test_collections-669', u'test_collections-641', u'test_collections-540', u'test_collections-660', u'test_collections-916', u'test_collections-837', u'test_collections-977', u'test_collections-978', u'test_collections-711', u'test_collections-599', u'test_collections-797', u'test_collections-754'] 2021-05-18 01:03:30,967 | test | WARNING | Thread-51906 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:wGqaXE6Y-41-401000:FZh4ROaTt7-41-402000 update failed keys from task: [u'test_collections-924', u'test_collections-669', u'test_collections-540', u'test_collections-660', u'test_collections-916', u'test_collections-837', u'test_collections-977', u'test_collections-978', u'test_collections-711', u'test_collections-599', u'test_collections-797', u'test_collections-754', u'test_collections-673', u'test_collections-590'] 2021-05-18 01:03:31,673 | test | WARNING | Thread-51849 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:GMyjIa1C8-41-395000 delete failed keys from task: [u'test_collections-37', u'test_collections-309', u'test_collections-467', u'test_collections-468', u'test_collections-268', u'test_collections-243', u'test_collections-365', u'test_collections-361', u'test_collections-480', u'test_collections-25', u'test_collections-419', u'test_collections-278', u'test_collections-334', u'test_collections-455', u'test_collections-299', u'test_collections-175', u'test_collections-274', u'test_collections-250'] 2021-05-18 01:03:42,513 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 01:04:33,732 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 01:04:33,733 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 01:04:56,903 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 01:04:56,903 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 01:05:20,565 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 01:05:20,565 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 01:06:16,845 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1319999695 seconds to complete 2021-05-18 01:06:16,859 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2400.021s OK During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 22 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_22 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_23 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=4,nodes_swap=2,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_23', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 23, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 01:06:26,256 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #23 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 01:06:26,944 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #23 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 01:06:27,433 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a299 type=client open_count=1 channel=[id: 0x59b328a3, 0.0.0.0/0.0.0.0:55506] timeout=300.0>) 2021-05-18 01:06:48,280 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a29a type=client open_count=1 channel=[id: 0x9c5b87a6, 0.0.0.0/0.0.0.0:53092] timeout=300.0>) 2021-05-18 01:06:50,301 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a29b type=client open_count=1 channel=[id: 0x4464d7bc, 0.0.0.0/0.0.0.0:60840] timeout=300.0>) 2021-05-18 01:06:52,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a29c type=client open_count=1 channel=[id: 0xa0b3fe5f, 0.0.0.0/0.0.0.0:56638] timeout=300.0>) 2021-05-18 01:06:54,359 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a29d type=client open_count=1 channel=[id: 0x29c07512, 0.0.0.0/0.0.0.0:60796] timeout=300.0>) 2021-05-18 01:06:56,381 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x68a29e type=client open_count=1 channel=[id: 0x0a407bea, 0.0.0.0/0.0.0.0:53771] timeout=300.0>) 2021-05-18 01:06:58,398 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x68a29f type=client open_count=1 channel=[id: 0x0daaa47d, 0.0.0.0/0.0.0.0:43858] timeout=300.0>) 2021-05-18 01:07:22,061 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x68a2a0 type=client open_count=1 channel=[id: 0x3fcec1ac, 0.0.0.0/0.0.0.0:56381] timeout=300.0>) 2021-05-18 01:07:24,079 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x68a2a1 type=client open_count=1 channel=[id: 0xa7461a5f, 0.0.0.0/0.0.0.0:50950] timeout=300.0>) 2021-05-18 01:07:26,101 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x68a2a2 type=client open_count=1 channel=[id: 0x56e64121, 0.0.0.0/0.0.0.0:52988] timeout=300.0>) 2021-05-18 01:07:28,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x68a2a3 type=client open_count=1 channel=[id: 0x236dd34e, 0.0.0.0/0.0.0.0:37869] timeout=300.0>) 2021-05-18 01:07:51,586 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x68a2a4 type=client open_count=1 channel=[id: 0xf34654b8, 0.0.0.0/0.0.0.0:38387] timeout=300.0>) 2021-05-18 01:07:53,605 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x68a2a5 type=client open_count=1 channel=[id: 0x5e520658, 0.0.0.0/0.0.0.0:45815] timeout=300.0>) 2021-05-18 01:07:55,624 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x68a2a6 type=client open_count=1 channel=[id: 0x217d8062, 0.0.0.0/0.0.0.0:56413] timeout=300.0>) 2021-05-18 01:07:57,641 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x68a2a7 type=client open_count=1 channel=[id: 0x10cd7b4a, 0.0.0.0/0.0.0.0:54423] timeout=300.0>) 2021-05-18 01:08:21,648 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x68a2a8 type=client open_count=1 channel=[id: 0x8f2265d9, 0.0.0.0/0.0.0.0:54489] timeout=300.0>) 2021-05-18 01:08:23,668 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x68a2a9 type=client open_count=1 channel=[id: 0x512dc3d6, 0.0.0.0/0.0.0.0:44475] timeout=300.0>) 2021-05-18 01:08:25,686 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x68a2aa type=client open_count=1 channel=[id: 0x24a09da9, 0.0.0.0/0.0.0.0:38433] timeout=300.0>) 2021-05-18 01:08:49,851 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x68a2ab type=client open_count=1 channel=[id: 0x4ca695c0, 0.0.0.0/0.0.0.0:45961] timeout=300.0>) 2021-05-18 01:08:51,869 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x68a2ac type=client open_count=1 channel=[id: 0x306205c0, 0.0.0.0/0.0.0.0:46041] timeout=300.0>) 2021-05-18 01:08:53,891 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x68a2ad type=client open_count=1 channel=[id: 0x237d4cf4, 0.0.0.0/0.0.0.0:54252] timeout=300.0>) 2021-05-18 01:08:55,910 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 01:09:12,627 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 01:09:26,426 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 01:09:26,450 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #23 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 01:09:26,494 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #23 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 01:09:36,891 | test | INFO | pool-25-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.59017950898 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 01:09:36,924 | test | INFO | pool-25-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 01:09:47,000 | test | INFO | pool-25-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 01:09:47,026 | test | INFO | pool-25-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.135999918 sec 2021-05-18 01:09:47,289 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 01:10:34,842 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 01:10:35,117 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 697776904 | 86004478 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 662098320 | 106718009 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 1018531984 | 57927867 | +---------+-----------+----------+------------+-----+-------+------------+------------+-----------+ 2021-05-18 01:10:35,118 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 01:11:16,012 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 01:11:16,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 01:30:22,973 | test | INFO | pool-25-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****...................................................................*........................****X | 16.495 | | 0.000 | ******************************************........................***********X | 4.499 | | 0.000 | ***********........................*****X | 4.500 | | 0.000 | *****************************************..*..****X | 0.801 | | 0.600 | ..******........*.....****X | 3.000 | | 1.900 | .........X | 1.900 | | 1.501 | .......****X | 1.501 | | 0.400 | .X | 0.400 | | 0.000 | *****......*..*****..................*.......****X | 6.002 | | 4.402 | .......................X | 4.402 | | 0.000 | ***********........*........................*****..........******..........................................................*****X | 18.004 | | 5.900 | ...............................*****.........*..........................****X | 12.103 | | 12.003 | .................................................................X | 12.003 | | 12.001 | .................................................................****X | 12.001 | | 6.406 | ..................................X | 6.406 | | 0.000 | ***********.......*.........****X | 3.000 | | 1.601 | .......X | 1.601 | | 0.000 | *****...............*...................*****............*...........****X | 10.301 | | 5.404 | .............................X | 5.404 | | 0.000 | ***********...*.....****X | 1.500 | | 0.801 | ...X | 0.801 | | 0.000 | *****************........*****...............*.......................*****....................*....****X | 12.702 | | 6.403 | ..................................X | 6.403 | | 0.000 | *****........*****X | 1.500 | | 1.500 | .......*****X | 1.500 | | 1.500 | .......**********X | 1.500 | | 1.500 | .......X | 1.500 | | 1.500 | .......****X | 1.500 | | 0.800 | ...X | 0.800 | | 0.000 | *****.........................................................................*..........................****X | 18.001 | | 15.901 | .......................................................................................X | 15.901 | | 13.501 | .........................................................................****X | 13.501 | | 9.900 | .....................................................X | 9.900 | | 0.000 | *****************.......*.........*****X | 3.001 | | 3.001 | ...............****X | 3.001 | | 1.601 | .......X | 1.601 | | 0.000 | ****.*..*.....***X | 1.500 | | 0.800 | ...X | 0.800 | | 0.000 | *****....*............*****..........*......*****...................................****X | 12.100 | | 9.401 | ...................................................X | 9.401 | | 0.000 | *****........*................................****X | 7.202 | | 6.601 | ...................................X | 6.601 | | 5.801 | ...............................X | 5.801 | | 4.800 | .........................***X | 4.800 | | 3.901 | ....................X | 3.901 | | 1.501 | .......X | 1.501 | | 0.000 | **********...*.....****X | 1.500 | | 0.800 | ...X | 0.800 | | 0.000 | ***********...........*..................****X | 5.300 | | 2.200 | ...........****X | 2.200 | | 2.200 | ...........*..*..***X | 3.000 | | 1.600 | .......X | 1.600 | | 0.000 | ****************..*......*****X | 1.500 | | 1.500 | .......*****..........................................*****X | 9.002 | | 4.803 | .........................X | 4.803 | | 0.000 | *****************....................................*.............****X | 9.000 | | 0.000 | *****...*.............****X | 3.000 | | 1.900 | .........X | 1.900 | | 1.500 | .......*****........******............*.....****X | 5.999 | | 2.700 | .............X | 2.700 | | 1.500 | .......****X | 1.500 | | 1.100 | .....X | 1.100 | | 0.000 | *****........*...................******...........****X | 7.000 | | 3.000 | ...............*****X | 3.000 | | 0.800 | ...X | 0.800 | | 0.000 | ****.*.......****X | 1.500 | | 1.200 | .....X | 1.200 | | 1.200 | .....*****....******..............*****X | 4.500 | | 2.400 | ............X | 2.400 | | 0.000 | *******************************************X | 0.000 | +--------+-----------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 01:30:23,316 | test | INFO | pool-25-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****..........................................................................*..........................****X | 820.647 | | 715.523 | ......................................................................................*****X | 715.523 | | 598.373 | .......................................................................X | 598.373 | | 488.399 | ..........................................................*****........*.****X | 561.250 | | 471.095 | ........................................................X | 471.095 | | 461.265 | .......................................................*****......******.....*..****X | 574.112 | | 513.566 | .............................................................X | 513.566 | | 492.563 | ...........................................................****X | 492.563 | | 477.350 | .........................................................X | 477.350 | | 471.230 | ........................................................*****.*..****X | 494.935 | | 421.960 | ..................................................****X | 421.960 | | 402.996 | ................................................X | 402.996 | | 394.672 | ...............................................*****..******.............****X | 523.724 | | 504.083 | ............................................................X | 504.083 | | 444.410 | .....................................................****X | 444.410 | | 442.107 | ....................................................*.****X | 443.739 | | 440.360 | ....................................................X | 440.360 | | 417.897 | .................................................*****......*****X | 466.194 | | 455.792 | ......................................................****X | 455.792 | | 426.922 | ...................................................X | 426.922 | | 421.702 | ..................................................*****.....X | 465.736 | | 461.614 | .......................................................****X | 461.614 | | 445.413 | .....................................................*****X | 447.746 | | 431.973 | ...................................................X | 431.973 | | 420.013 | ..................................................****X | 420.013 | | 389.400 | ..............................................******........*****X | 454.461 | | 398.233 | ...............................................*****....X | 431.089 | | 424.045 | ..................................................*****..*...****X | 460.903 | | 449.203 | .....................................................X | 449.203 | | 439.606 | ....................................................*****..X | 454.170 | | 443.894 | .....................................................****X | 443.894 | | 440.995 | ....................................................*****X | 440.995 | | 307.733 | ....................................******............****X | 409.128 | | 372.242 | ............................................X | 372.242 | | 288.615 | ..................................*****...*.......*****..........******.....*****X | 494.202 | | 442.938 | ....................................................******...*****X | 464.218 | | 312.507 | .....................................******......******......****X | 411.767 | | 410.506 | .................................................*..*****...X | 454.422 | | 446.000 | .....................................................****X | 446.000 | | 431.115 | ...................................................*...****X | 455.289 | | 438.213 | ....................................................X | 438.213 | | 395.911 | ...............................................****X | 395.911 | | 392.544 | ..............................................*....*****...****X | 445.125 | | 382.837 | .............................................X | 382.837 | | 349.733 | .........................................***********........*......*****.X | 472.820 | | 462.983 | .......................................................*****...*..****X | 504.822 | | 468.332 | ........................................................X | 468.332 | | 447.719 | .....................................................****X | 447.719 | | 435.322 | ....................................................X | 435.322 | | 390.692 | ..............................................*****..*....*****..*.****X | 462.830 | | 319.499 | .....................................******...******.............******........****X | 510.732 | | 482.477 | .........................................................X | 482.477 | | 422.789 | ..................................................*****..X | 440.776 | | 419.107 | ..................................................****X | 419.107 | | 362.682 | ...........................................*.*****.......******.X | 441.617 | | 440.240 | ....................................................***X | 440.240 | | 421.331 | ..................................................X | 421.331 | | 419.702 | ..................................................****X | 419.702 | | 386.278 | ..............................................X | 386.278 | | 338.165 | ........................................*****.....*.......****X | 436.249 | | 392.496 | ..............................................X | 392.496 | | 387.993 | ..............................................****....*.........****X | 493.621 | | 442.303 | ....................................................X | 442.303 | | 400.297 | ...............................................****X | 400.297 | | 384.339 | .............................................X | 384.339 | | 362.517 | ...........................................*****.....*............****X | 503.199 | | 403.101 | ................................................******..........****X | 486.370 | | 439.415 | ....................................................X | 439.415 | | 437.566 | ....................................................****X | 437.566 | | 427.656 | ...................................................X | 427.656 | | 348.997 | .........................................****X | 348.997 | | 345.917 | .........................................*....*****.....*****..X | 438.564 | | 427.975 | ...................................................*****....*.....****X | 501.138 | | 437.473 | ....................................................*****X | 437.473 | | 388.112 | ..............................................*****X | 388.112 | | 378.781 | .............................................******...*****X | 409.990 | | 350.277 | .........................................******.............****X | 456.712 | | 438.394 | ....................................................X | 438.394 | | 424.178 | ..................................................*****X | 424.178 | | 403.940 | ................................................*****X | 404.160 | | 399.489 | ...............................................*****....*****X | 430.434 | | 427.144 | ...................................................*****X | 427.144 | | 398.062 | ...............................................*****.....*...****X | 465.778 | | 448.974 | .....................................................X | 448.974 | | 443.009 | ....................................................*****.....*....****X | 510.821 | | 457.312 | ......................................................X | 457.312 | | 444.799 | .....................................................****X | 444.799 | | 378.638 | .............................................X | 378.638 | | 357.245 | ..........................................*****...........*...****X | 469.892 | | 388.674 | ..............................................X | 388.674 | | 369.406 | ............................................*****.*......*****..*****..X | 461.946 | | 454.413 | ......................................................***********X | 454.428 | | 431.085 | ...................................................******.....****X | 470.900 | | 460.389 | .......................................................X | 460.389 | | 367.182 | ...........................................*****...*............****X | 486.434 | | 424.855 | ..................................................X | 424.855 | | 402.374 | ................................................****X | 402.374 | | 402.049 | ...............................................******................******....*****X | 560.386 | | 484.899 | ..........................................................X | 484.899 | | 400.979 | ...............................................**X | 400.979 | +---------+-----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:30:23,657 | test | INFO | pool-25-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *****.....................................*..............****X | 13.496 | | 0.000 | ************************************************..*...****X | 1.500 | | 0.000 | **********************************..*...****X | 1.500 | | 0.000 | ************************...*........****X | 3.001 | | 3.000 | ..........X | 3.000 | | 3.000 | ..........****X | 3.000 | | 1.501 | ....******.....*****X | 2.800 | | 0.200 | *****.....*............****X | 4.501 | | 3.301 | ...........X | 3.301 | | 0.000 | ***********........*...****X | 3.000 | | 1.602 | .....X | 1.602 | | 0.000 | *****.....*......****X | 3.001 | | 2.200 | .......X | 2.200 | | 0.000 | *****................*..................****X | 9.002 | | 8.202 | ..............................X | 8.202 | | 6.000 | .....................****X | 6.000 | | 0.000 | ************************.......*.....................*****.................*****X | 12.001 | | 8.801 | ................................X | 8.801 | | 0.000 | ***********************.*....*****......*.................****X | 7.500 | | 1.500 | ....*****........*.........****X | 5.999 | | 5.302 | ...................X | 5.302 | | 4.501 | ................****X | 4.501 | | 0.000 | ******........*...****X | 3.001 | | 0.801 | ..X | 0.801 | | 0.000 | ***********............................*****X | 7.501 | | 3.000 | ..........****X | 3.000 | | 1.601 | .....X | 1.601 | | 0.000 | *****....................................................................*****X | 18.001 | | 13.501 | ..................................................*****X | 13.501 | | 0.000 | *****************...*......*****.............*....****X | 7.002 | | 1.100 | ...*****.*.*.....****.....*........*****...................................*.....................*....................***X | 26.198 | | 22.299 | ....................................................................................X | 22.299 | | 5.001 | ..................******...****X | 6.000 | | 4.400 | ...............X | 4.400 | | 0.000 | ***********..........*............****X | 6.000 | | 3.201 | ...........X | 3.201 | | 0.000 | *****............................*****X | 7.500 | | 6.000 | .....................****X | 6.000 | | 4.800 | .................X | 4.800 | | 0.000 | ***********....*......*............****X | 6.000 | | 1.500 | ....*****....*.........*....***X | 5.999 | | 3.200 | ...........X | 3.200 | | 0.000 | ****************......*................*****.........*...*****...................................................*****X | 22.506 | | 0.000 | *****************............*.....****X | 4.500 | | 0.000 | *****........*.........****X | 4.500 | | 3.400 | ...........X | 3.400 | | 3.000 | ..........****X | 3.000 | | 1.500 | ....******......*****X | 2.999 | | 0.000 | ************.................*****X | 4.499 | | 0.000 | ******......................*****X | 6.000 | | 0.000 | *****..*...*****....*..****X | 2.999 | | 0.800 | ..X | 0.800 | | 0.000 | *****.*....****X | 1.500 | | 1.500 | ....*****X | 1.500 | | 0.800 | ..X | 0.800 | | 0.000 | **************************************X | 0.000 | +--------+----------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 01:30:23,934 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 78.9507069291 | 6437847040 | 4112261120 | 2599841792 / 7511588864 | 126025 / 252835 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 65.077830739 | 6437847040 | 4150087680 | 2545364992 / 7511588864 | 125582 / 254792 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 62.5740756197 | 6437847040 | 3751653376 | 3099463680 / 9530753024 | 126399 / 247666 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 69.0371827136 | 6437847040 | 4225613824 | 2513920000 / 7511588864 | 126426 / 253560 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 01:30:35,709 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 01:31:47,773 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 161993568 | 254789164 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 159023920 | 285094643 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 440317440 | 449236563 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 01:31:47,773 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #23 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 01:31:56,246 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during swap_rebalance 2021-05-18 01:31:56,246 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-18 01:32:00,651 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 01:32:01,033 | test | INFO | pool-25-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 11.3033333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 17.8423582827 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 17.8919648661 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 9.14333333333 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 01:32:01,125 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 01:32:06,311 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 01:32:09,839 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 01:32:11,421 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.44 2021-05-18 01:32:16,516 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 1.56 2021-05-18 01:32:21,637 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.9 2021-05-18 01:32:26,720 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 4.54 2021-05-18 01:32:31,865 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 6.22 2021-05-18 01:32:37,022 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 8.23 2021-05-18 01:32:42,151 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 10.74 2021-05-18 01:32:47,286 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 12.82 2021-05-18 01:32:52,398 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 15.1 2021-05-18 01:32:55,512 | test | INFO | pool-25-thread-17 | [table_view:display:72] Keys failed in bucket1:2Yu7WUt04vly4cMW-7-243000:Uk3CUVjZvVvlKNjgFii-7-359000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:55,512 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:55,513 | test | INFO | pool-25-thread-17 | [table_view:display:72] Keys failed in bucket1:2Yu7WUt04vly4cMW-7-243000:Uk3CUVjZvVvlKNjgFii-7-359000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:55,513 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:55,516 | test | INFO | pool-25-thread-29 | [table_view:display:72] Keys failed in bucket1:2Yu7WUt04vly4cMW-7-243000:VqnqEKmy4WapJB1Ap-7-359000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:55,516 | test | INFO | pool-25-thread-29 | [table_view:display:72] Keys failed in bucket1:2Yu7WUt04vly4cMW-7-243000:VqnqEKmy4WapJB1Ap-7-359000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-245 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:32:57,512 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 17.01 2021-05-18 01:33:02,622 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 18.67 2021-05-18 01:33:07,736 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 20.39 2021-05-18 01:33:12,848 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 22.1 2021-05-18 01:33:17,936 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 23.46 2021-05-18 01:33:23,030 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 25.12 2021-05-18 01:33:28,128 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 26.62 2021-05-18 01:33:30,194 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-711 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-681 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:30,196 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-606 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-711 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-796 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-681 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:33,213 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 28.34 2021-05-18 01:33:38,325 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.08 2021-05-18 01:33:39,697 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-846 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:39,698 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-846 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:43,448 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 32.03 2021-05-18 01:33:48,533 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 01:33:49,309 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:49,309 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:33:53,611 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 01:33:58,779 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 01:34:00,947 | test | INFO | pool-25-thread-15 | [table_view:display:72] Keys failed in bucket2:_default:RoMY5-47-336000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:34:04,019 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 34.9 2021-05-18 01:34:09,184 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.2 2021-05-18 01:34:14,290 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.1 2021-05-18 01:34:19,438 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.95 2021-05-18 01:34:24,555 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 42.58 2021-05-18 01:34:29,651 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 44.09 2021-05-18 01:34:34,813 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 45.56 2021-05-18 01:34:39,940 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 47.55 2021-05-18 01:34:45,073 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 49.19 2021-05-18 01:34:48,128 | test | INFO | pool-25-thread-6 | [table_view:display:72] Keys failed in bucket2:OjhqHq3XV-6-297000:vM6Hq0Qyl2-6-629000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2620 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:34:48,130 | test | INFO | pool-25-thread-6 | [table_view:display:72] Keys failed in bucket2:OjhqHq3XV-6-297000:vM6Hq0Qyl2-6-629000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2620 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:34:48,357 | test | INFO | pool-25-thread-10 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:IhDmPlC48%C_-9-151000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:34:48,358 | test | INFO | pool-25-thread-10 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:IhDmPlC48%C_-9-151000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-248 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:34:50,197 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 50.88 2021-05-18 01:34:55,279 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 51.81 2021-05-18 01:35:00,387 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 53.36 2021-05-18 01:35:05,536 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 54.56 2021-05-18 01:35:10,638 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 55.98 2021-05-18 01:35:15,756 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 57.19 2021-05-18 01:35:16,786 | test | INFO | pool-25-thread-25 | [table_view:display:72] Keys failed in bucket2:_default:BanyxcFj6PkTQk4WeMM-47-336000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:20,371 | test | INFO | pool-25-thread-11 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:1Z4y-qckzt0SvAAZBS-5-470000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:20,372 | test | INFO | pool-25-thread-11 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:1Z4y-qckzt0SvAAZBS-5-470000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1244 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:20,848 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 58.37 2021-05-18 01:35:25,943 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 59.84 2021-05-18 01:35:31,069 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 61.53 2021-05-18 01:35:36,200 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 63.16 2021-05-18 01:35:41,296 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 64.64 2021-05-18 01:35:46,381 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.4 2021-05-18 01:35:46,668 | test | INFO | pool-25-thread-28 | [table_view:display:72] Keys failed in bucket2:JdyfE-5-880000:2UwBsUidLk4L-6-42000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:46,668 | test | INFO | pool-25-thread-18 | [table_view:display:72] Keys failed in bucket2:OjhqHq3XV-6-297000:7QFrBLiMt3ZudDtWLz-6-628000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:46,668 | test | INFO | pool-25-thread-28 | [table_view:display:72] Keys failed in bucket2:JdyfE-5-880000:2UwBsUidLk4L-6-42000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:46,670 | test | INFO | pool-25-thread-18 | [table_view:display:72] Keys failed in bucket2:OjhqHq3XV-6-297000:7QFrBLiMt3ZudDtWLz-6-628000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1941 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:47,292 | test | INFO | pool-25-thread-11 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:1Z4y-qckzt0SvAAZBS-5-470000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:47,292 | test | INFO | pool-25-thread-11 | [table_view:display:72] Keys failed in bucket2:vFg6Zklocb_d16Soap-5-220000:1Z4y-qckzt0SvAAZBS-5-470000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1415 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1585 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:35:51,483 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 01:35:56,576 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 01:36:03,193 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.77 2021-05-18 01:36:08,315 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 68.09 2021-05-18 01:36:13,401 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 69.18 2021-05-18 01:36:18,494 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.61 2021-05-18 01:36:20,013 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:_default:EjcAJFKtIylxf8s66-47-365000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-623 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-657 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-687 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-666 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:20,013 | test | INFO | pool-25-thread-13 | [table_view:display:72] Keys failed in default:scope2:7Q_vGMN7VVgAY_-47-373000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-528 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-516 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-694 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:21,904 | test | INFO | pool-25-thread-9 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:7YmIAiuKc8pBlAgfP-47-375000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-704 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-867 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-822 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-896 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:23,605 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 71.46 2021-05-18 01:36:28,707 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 72.98 2021-05-18 01:36:33,809 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 74.32 2021-05-18 01:36:34,680 | test | INFO | pool-25-thread-15 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:o-47-376000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-934 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:37,464 | test | INFO | pool-25-thread-13 | [table_view:display:72] Keys failed in default:scope2:7Q_vGMN7VVgAY_-47-373000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-849 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:38,572 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:_default:EjcAJFKtIylxf8s66-47-365000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-708 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-729 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-757 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-834 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-712 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-798 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:38,904 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 75.75 2021-05-18 01:36:41,078 | test | INFO | pool-25-thread-8 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:foOziQFj-47-374000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-548 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-668 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-698 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-682 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:43,994 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 76.87 2021-05-18 01:36:46,107 | test | INFO | pool-25-thread-3 | [table_view:display:72] Keys failed in default:scope1:bwlRB5Osu-47-367000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-608 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-626 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-555 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-633 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-644 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:46,323 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:QIT-8-919000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1341 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1342 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:46,325 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:QIT-8-919000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1341 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1342 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:49,099 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 77.98 2021-05-18 01:36:50,334 | test | INFO | pool-25-thread-9 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:7YmIAiuKc8pBlAgfP-47-375000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-978 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:52,082 | test | INFO | pool-25-thread-26 | [table_view:display:72] Keys failed in default:scope2:6-9-152000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2977 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:52,082 | test | INFO | pool-25-thread-26 | [table_view:display:72] Keys failed in default:scope2:6-9-152000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2977 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:54,213 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 78.84 2021-05-18 01:36:55,305 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:_default:EjcAJFKtIylxf8s66-47-365000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:55,723 | test | INFO | pool-25-thread-13 | [table_view:display:72] Keys failed in default:scope2:7Q_vGMN7VVgAY_-47-373000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-915 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-956 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:36:59,312 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 80.03 2021-05-18 01:37:04,427 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 81.22 2021-05-18 01:37:04,654 | test | INFO | pool-25-thread-15 | [table_view:display:72] Keys failed in default:Y-47-376000:Oe3MhI-47-379000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-509 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-556 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:07,911 | test | INFO | pool-25-thread-3 | [table_view:display:72] Keys failed in default:scope1:bwlRB5Osu-47-367000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-889 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:07,911 | test | INFO | pool-25-thread-8 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:foOziQFj-47-374000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-827 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-889 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-885 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-840 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:09,506 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 82.58 2021-05-18 01:37:14,631 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 83.81 2021-05-18 01:37:19,760 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 85.19 2021-05-18 01:37:24,868 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 86.61 2021-05-18 01:37:29,334 | test | INFO | pool-25-thread-25 | [table_view:display:72] Keys failed in default:_default:7422hX-47-365000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-513 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:29,973 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 87.94 2021-05-18 01:37:33,789 | test | INFO | pool-25-thread-8 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:foOziQFj-47-374000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-949 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:33,792 | test | INFO | pool-25-thread-3 | [table_view:display:72] Keys failed in default:scope1:bwlRB5Osu-47-367000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-939 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-937 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-905 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-949 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-955 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:35,107 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 89.23 2021-05-18 01:37:35,194 | test | INFO | pool-25-thread-15 | [table_view:display:72] Keys failed in default:Y-47-376000:Oe3MhI-47-379000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-819 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-836 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-810 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-773 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-895 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-730 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-893 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:40,229 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 90.58 2021-05-18 01:37:40,992 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in default:scope2:vs53nd379nT-47-373000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-515 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-559 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-625 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-599 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-545 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:45,372 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 91.88 2021-05-18 01:37:46,345 | test | INFO | pool-25-thread-30 | [table_view:display:72] Keys failed in default:8ryw-3K-8-447000:xRP_0CDZ_0-8-584000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2707 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2706 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2697 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:46,346 | test | INFO | pool-25-thread-30 | [table_view:display:72] Keys failed in default:8ryw-3K-8-447000:xRP_0CDZ_0-8-584000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2707 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2706 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2697 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:37:50,565 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 93.22 2021-05-18 01:37:55,687 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 94.6 2021-05-18 01:38:00,818 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 96.11 2021-05-18 01:38:01,661 | test | INFO | pool-25-thread-9 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:rTyRix6j-V-47-375000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:01,661 | test | INFO | pool-25-thread-13 | [table_view:display:72] Keys failed in default:scope2:KZMNUjXe7B92z-47-373000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-69 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-76 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-98 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-33 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-135 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-165 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-170 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:03,069 | test | INFO | pool-25-thread-24 | [table_view:display:72] Keys failed in default:bD3pUNQLCQDk-8-89000:OjuZnwMMhH-uP8NY_t%t-8-280000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3438 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:03,069 | test | INFO | pool-25-thread-24 | [table_view:display:72] Keys failed in default:bD3pUNQLCQDk-8-89000:OjuZnwMMhH-uP8NY_t%t-8-280000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3438 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:05,934 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 97.55 2021-05-18 01:38:06,657 | test | INFO | pool-25-thread-25 | [table_view:display:72] Keys failed in default:_default:7422hX-47-365000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-800 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-864 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:08,101 | test | INFO | pool-25-thread-20 | [table_view:display:72] Keys failed in default:scope2:vs53nd379nT-47-373000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-818 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-859 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-735 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-769 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-821 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-872 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-784 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-794 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-770 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:08,184 | test | INFO | pool-25-thread-22 | [table_view:display:72] Keys failed in default:9x75xAfHFO4Ub-47-373000:BxFBjUuOorRVL9N0kpE-47-375000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-44 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-117 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-84 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-133 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-176 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-152 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-192 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:11,038 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 98.79 2021-05-18 01:38:16,148 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 99.91 2021-05-18 01:38:18,908 | test | INFO | pool-25-thread-6 | [table_view:display:72] Keys failed in default:scope2:collection2 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-25 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-92 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-144 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-166 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-101 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-123 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-7 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:20,405 | test | INFO | pool-25-thread-13 | [table_view:display:72] Keys failed in default:scope2:KZMNUjXe7B92z-47-373000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-316 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-206 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-346 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-347 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-303 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-215 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-353 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-243 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-385 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-286 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-396 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-371 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-250 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | | test_collections-293 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:20,726 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:QIT-8-919000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:20,727 | test | INFO | pool-25-thread-14 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:QIT-8-919000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1970 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:21,243 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 01:38:25,193 | test | INFO | pool-25-thread-21 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:SH70jsnwF5oQWiWb_0Xl-8-920000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:25,194 | test | INFO | pool-25-thread-21 | [table_view:display:72] Keys failed in default:8xx7J4_-HC_0-8-788000:SH70jsnwF5oQWiWb_0Xl-8-920000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3732 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:26,313 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 01:38:31,365 | test | INFO | pool-25-thread-5 | [table_view:display:72] Keys failed in default:scope1:xuJ-47-367000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-607 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-629 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-642 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:31,368 | test | INFO | pool-25-thread-29 | [table_view:display:72] Keys failed in default:scope1:collection_1 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-618 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-614 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-648 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-522 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-578 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-567 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-531 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-651 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-574 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-593 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-692 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 01:38:36,430 | test | INFO | pool-25-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 01:38:38,671 | test | INFO | pool-25-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 397.638000011 sec 2021-05-18 01:40:59,512 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:40:59,513 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:40:59,515 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:40:59,516 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:40:59,516 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:40:59,517 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 7 docs from 0 to 5000 2021-05-18 01:40:59,519 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:41:49,720 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-18 01:41:49,720 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:41:49,721 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:41:49,723 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-18 01:41:49,724 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-18 01:41:49,730 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 01:42:46,575 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 3 docs from 0 to 5000 2021-05-18 01:42:46,576 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 01:42:46,581 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-18 01:43:47,351 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 5 docs from 500 to 980 2021-05-18 01:43:47,354 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 13 docs from 500 to 980 2021-05-18 01:43:47,355 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-18 01:43:47,355 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 16 docs from 500 to 980 2021-05-18 01:43:47,357 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-18 01:43:47,358 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-18 01:43:47,358 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-18 01:43:47,372 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 14 docs from 500 to 980 2021-05-18 01:43:47,375 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 10 docs from 0 to 500 2021-05-18 01:43:47,375 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 24 docs from 0 to 500 2021-05-18 01:43:47,375 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 10 docs from 0 to 500 2021-05-18 01:43:47,375 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 11 docs from 500 to 980 2021-05-18 01:43:47,375 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 8 docs from 0 to 500 2021-05-18 01:43:47,377 | test | ERROR | pool-25-thread-16 | [task:call:3809] Failed to load 9 docs from 500 to 980 2021-05-18 01:52:57,576 | test | INFO | pool-25-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..*****.......****.........****..........................***X | 55.497 | | 38.098 | .............................X | 38.098 | | 28.499 | ......................***.....*....................***X | 60.404 | | 43.899 | ..................................***X | 43.899 | | 34.092 | ..........................X | 34.092 | | 30.692 | .......................****............***X | 44.816 | | 44.496 | ...................................******................................................................***X | 123.516 | | 74.744 | ...........................................................****X | 74.744 | | 66.354 | ....................................................***X | 66.354 | | 62.189 | .................................................*X | 62.189 | | 62.189 | .................................................*X | 62.189 | | 62.189 | .................................................*.....****X | 68.596 | | 62.710 | .................................................***X | 62.710 | | 58.900 | ..............................................*..****...X | 64.280 | | 60.784 | ................................................****............................***X | 96.045 | | 81.098 | ................................................................****X | 81.098 | | 51.393 | ........................................*****...........................****X | 84.608 | | 83.711 | ..................................................................*****.........***X | 94.175 | | 88.975 | .......................................................................X | 88.975 | | 55.308 | ...........................................****.............*..................***X | 93.486 | | 88.650 | ......................................................................****X | 88.650 | | 62.582 | .................................................*****.......****............*...***X | 89.233 | | 42.502 | .................................X | 42.502 | | 24.698 | ..................****......***X | 31.297 | | 28.297 | .....................*****.....****X | 34.191 | | 26.011 | ....................***X | 26.011 | | 0.200 | ****X | 0.200 | | 0.000 | *****************.................******....................*****X | 46.300 | | 39.204 | ..............................X | 39.204 | | 33.504 | ..........................*****.......*****X | 42.001 | | 38.099 | .............................******...***********X | 41.394 | | 40.110 | ...............................*.*****..*****X | 43.499 | | 38.491 | ..............................*****X | 38.491 | | 37.204 | .............................******..*.****X | 41.495 | | 40.208 | ...............................*****X | 40.208 | | 30.703 | .......................X | 30.703 | | 10.398 | .......*****....................................******..*******...*****X | 61.000 | | 48.100 | .....................................******......******...............*****X | 73.085 | | 52.500 | .........................................******......****X | 60.502 | | 54.202 | ..........................................X | 54.202 | | 40.197 | ...............................*****X | 40.197 | | 0.000 | ******..***X | 3.000 | | 3.000 | .*X | 3.000 | | 2.000 | *****...X | 5.400 | | 5.199 | ...****X | 5.199 | | 1.400 | ******X | 1.400 | | 1.100 | *****X | 1.100 | | 0.400 | *****X | 0.400 | | 0.000 | ******..*****X | 2.601 | | 0.200 | ***********.***********X | 2.200 | | 0.200 | *******.****X | 1.300 | | 0.000 | ***********X | 0.600 | | 0.200 | ******..*****X | 3.299 | | 1.800 | X | 1.800 | | 1.160 | ****X | 1.160 | | 1.059 | *****X | 1.102 | | 1.002 | X | 1.002 | | 0.901 | ***********.******..*****X | 4.005 | | 0.800 | *****X | 0.800 | | 0.600 | *..****X | 2.703 | | 1.703 | X | 1.703 | | 1.302 | ****X | 1.302 | | 0.702 | X | 0.702 | | 0.000 | ************...*****X | 3.804 | | 0.601 | *****X | 0.601 | | 0.401 | ************.*****X | 2.102 | | 1.001 | ******.*****X | 1.401 | | 1.001 | X | 1.001 | | 0.901 | *****.*****X | 2.402 | | 1.602 | X | 1.602 | | 0.000 | *****.*..****X | 4.102 | | 1.202 | X | 1.202 | | 0.600 | *****************X | 1.100 | | 0.900 | ******.*****.*.*****X | 4.501 | | 1.501 | X | 1.501 | | 0.700 | *****.***********X | 2.400 | | 1.100 | X | 1.100 | | 0.800 | *****.X | 2.000 | | 2.000 | ****X | 2.000 | | 1.601 | ******.*****X | 2.601 | | 1.301 | X | 1.301 | | 1.000 | *****.*****X | 2.001 | | 0.800 | ***********.******X | 1.901 | | 1.600 | ******..****X | 4.101 | | 1.500 | X | 1.500 | | 1.200 | ****X | 1.200 | | 0.000 | X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:52:57,624 | test | INFO | pool-25-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +---------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...........................................................................X | 678.652 | | 678.652 | ..........................................................................**X | 678.652 | | 449.074 | ................................................X | 449.074 | | 140.726 | ..............****...****............***X | 273.487 | | 192.061 | ....................X | 192.061 | | 187.689 | ...................**X | 187.689 | | 130.419 | .............****.X | 144.301 | | 123.495 | ............****.............................................***X | 528.379 | | 344.947 | .....................................X | 344.947 | | 157.834 | ................****.........*****..............................*............**X | 621.383 | | 439.176 | ...............................................X | 439.176 | | 336.898 | ....................................***...............*................................................**X | 903.508 | | 903.508 | ...................................................................................................X | 903.508 | | 221.193 | .......................***X | 221.193 | | 162.202 | ................X | 162.202 | | 149.181 | ...............*X | 149.181 | | 149.181 | ...............*X | 149.181 | | 130.306 | .............X | 130.306 | | 119.195 | ............****...****X | 151.719 | | 123.889 | ............****X | 123.889 | | 86.917 | ........****.........................................*..............***X | 580.966 | | 427.908 | ..............................................X | 427.908 | | 286.810 | ..............................***X | 286.810 | | 252.684 | ..........................*****..***X | 270.957 | | 251.866 | ..........................X | 251.866 | | 187.953 | ...................***X | 187.953 | | 175.314 | ..................X | 175.314 | | 152.691 | ...............****.....*......***X | 245.281 | | 123.955 | ............****....****.....****X | 206.891 | | 174.313 | ..................X | 174.313 | | 157.073 | ................***.*.***X | 178.517 | | 120.944 | ............***X | 120.944 | | 115.645 | ...........X | 115.645 | | 73.674 | .......****.....****X | 117.644 | | 95.535 | .........****X | 96.035 | | 67.173 | ......***X | 67.173 | | 66.073 | ......*.....****X | 115.806 | | 99.704 | ..........X | 99.704 | | 85.625 | ........*****.......................*****X | 291.200 | | 199.980 | .....................*****X | 199.980 | | 156.900 | ................*****X | 156.900 | | 137.716 | ..............*****X | 137.716 | | 130.505 | .............******...................*****.*..*****X | 331.645 | | 167.275 | .................**X | 167.275 | | 167.275 | .................***.*X | 174.327 | | 174.327 | ..................**X | 174.327 | | 174.327 | ..................X | 174.327 | | 116.496 | ...........******........******..******.*****X | 216.671 | | 189.938 | ....................******..........****X | 282.942 | | 282.942 | ..............................*...............*****X | 416.099 | | 317.237 | ..................................******......*****X | 378.199 | | 256.600 | ...........................******.....******.******......*****X | 362.800 | | 209.207 | ......................******............******......*............*****........*............****X | 668.180 | | 543.727 | ...........................................................*****X | 543.727 | | 465.854 | ..................................................*****X | 465.854 | | 254.743 | ...........................*****X | 254.743 | | 237.876 | .........................******.....*****X | 287.610 | | 269.902 | ............................X | 269.902 | | 206.855 | .....................*****.............******..........*****X | 411.599 | | 206.887 | .....................*****........*X | 271.806 | | 271.806 | .............................**X | 271.806 | | 271.806 | .............................**...........***X | 375.199 | | 375.199 | ........................................X | 375.199 | | 347.489 | .....................................X | 347.489 | | 316.631 | ..................................******.......................****X | 526.846 | | 443.829 | ................................................X | 443.829 | | 414.743 | ............................................******....*****X | 444.770 | | 392.309 | ..........................................X | 392.309 | | 329.900 | ...................................*****...................*****X | 499.910 | | 356.881 | ......................................******.....*****X | 400.811 | | 364.186 | .......................................X | 364.186 | | 336.310 | ....................................****X | 336.310 | | 333.022 | ...................................*.**********X | 339.953 | | 315.432 | .................................******....*****X | 345.267 | | 320.887 | ..................................******...******..........*****X | 441.531 | | 381.749 | .........................................*****X | 381.749 | | 328.434 | ...................................******...******...*****X | 383.528 | | 346.560 | .....................................****X | 346.560 | | 331.526 | ...................................X | 331.526 | | 310.666 | .................................******........*****X | 385.910 | | 350.730 | .....................................*****X | 350.730 | | 320.910 | ..................................X | 320.910 | | 261.321 | ...........................*****.........******.......*****X | 398.441 | | 305.461 | ................................******........*****X | 376.451 | | 342.341 | ....................................******..*****X | 353.922 | | 314.418 | .................................*****.....*.......****X | 417.437 | | 413.396 | ............................................X | 413.396 | | 409.977 | ............................................****X | 409.977 | | 386.019 | .........................................X | 386.019 | | 349.434 | .....................................*****........*..****X | 440.473 | | 430.973 | ..............................................X | 430.973 | | 423.067 | .............................................****X | 423.067 | | 396.049 | ..........................................X | 396.049 | | 367.036 | .......................................****X | 367.036 | | 357.745 | ......................................X | 357.745 | | 355.859 | ......................................*****...*..****X | 403.803 | | 388.739 | ..........................................X | 388.739 | | 327.167 | ...................................******..........******...*****X | 450.694 | | 423.734 | .............................................*****X | 423.734 | | 417.780 | .............................................*****X | 417.780 | | 371.183 | ........................................******.......****X | 435.441 | | 430.075 | ..............................................X | 430.075 | | 428.344 | ..............................................*****......*..****X | 504.901 | | 438.119 | ...............................................X | 438.119 | +---------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:52:57,904 | test | INFO | pool-25-thread-27 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..***X | 3.200 | | 2.100 | *.***........*...***.*.....****..................***X | 39.898 | | 36.006 | ................................***X | 36.006 | | 33.500 | ..............................X | 33.500 | | 25.999 | .......................****................................***X | 60.386 | | 60.012 | ......................................................*****.............*****............................***X | 104.015 | | 87.010 | ...............................................................................X | 87.010 | | 67.777 | .............................................................****......****X | 73.523 | | 57.297 | ....................................................***X | 57.297 | | 49.804 | .............................................X | 49.804 | | 47.893 | ...........................................****......***X | 54.700 | | 51.103 | ..............................................X | 51.103 | | 49.092 | ............................................X | 49.092 | | 47.298 | ..........................................**X | 47.298 | | 44.695 | ........................................X | 44.695 | | 44.539 | ........................................***X | 44.539 | | 43.715 | .......................................****X | 43.715 | | 42.581 | ......................................***X | 42.581 | | 29.985 | ..........................X | 29.985 | | 23.295 | ....................****.......................*****..............X | 62.605 | | 62.107 | ........................................................****.......****X | 69.983 | | 56.211 | ...................................................****X | 56.211 | | 47.993 | ...........................................*****...........................****..............***X | 91.758 | | 67.305 | .............................................................****X | 67.305 | | 51.820 | ...............................................*****...............................****X | 85.590 | | 65.995 | ............................................................***X | 65.995 | | 62.391 | ........................................................X | 62.391 | | 53.996 | .................................................****...*.............***X | 72.080 | | 47.817 | ...........................................****..................*****................******..................*****....*****X | 107.800 | | 82.993 | ...........................................................................******.....*........****X | 96.100 | | 70.108 | ................................................................******...............*****X | 86.603 | | 65.297 | ...........................................................******.******......*****X | 73.189 | | 46.509 | ..........................................******X | 46.509 | | 36.099 | ................................******.....*****X | 41.490 | | 40.204 | ....................................*****X | 40.204 | | 39.294 | ...................................******..*****X | 42.008 | | 41.404 | .....................................*..****X | 43.391 | | 0.200 | ************...*****X | 3.500 | | 3.500 | ..X | 3.500 | | 1.800 | *****X | 1.800 | | 0.700 | *****X | 0.700 | | 0.000 | ******************.*****X | 1.200 | | 0.400 | ******.*****X | 2.000 | | 1.900 | *****.................******......................................*****X | 60.810 | | 59.794 | ......................................................******.*...****X | 64.402 | | 61.800 | ........................................................X | 61.800 | | 58.686 | .....................................................X | 58.686 | | 58.686 | .....................................................****.....*****X | 63.811 | | 62.900 | .........................................................*X | 62.900 | | 62.900 | .........................................................***X | 62.900 | | 60.196 | ......................................................***********X | 60.302 | | 0.600 | *****X | 0.600 | | 0.200 | *******...****X | 3.400 | | 2.399 | .*****X | 2.399 | | 2.001 | *****X | 2.001 | | 0.500 | *****X | 0.500 | | 0.300 | ******.*****X | 1.839 | | 1.839 | ******.*.****X | 3.305 | | 3.105 | .X | 3.105 | | 2.604 | .*****X | 2.604 | | 1.802 | *****X | 1.802 | | 0.901 | *****X | 0.901 | | 0.301 | X | 0.301 | | 0.200 | *****....*X | 4.807 | | 4.807 | ...***X | 4.807 | | 1.602 | *****X | 1.602 | | 1.001 | *****X | 1.001 | | 0.300 | ***********X | 0.400 | | 0.200 | ******.******.******.*****X | 3.303 | | 1.302 | X | 1.302 | | 0.700 | ****X | 0.700 | | 0.400 | X | 0.400 | | 0.200 | ******.******.*****X | 2.802 | | 0.000 | ******.*****X | 1.401 | | 0.000 | ******.*****************X | 1.901 | | 0.300 | ******...*****X | 3.801 | | 2.201 | .X | 2.201 | | 1.700 | ****X | 1.700 | | 0.501 | X | 0.501 | | 0.000 | ***********..*****X | 2.800 | | 1.500 | X | 1.500 | | 1.300 | **********X | 1.601 | | 1.300 | X | 1.300 | | 1.100 | ****X | 1.100 | | 0.800 | X | 0.800 | | 0.800 | *****.X | 1.601 | | 1.300 | *****.*****X | 2.701 | | 0.400 | ******.*****X | 1.800 | | 0.700 | ******..*****X | 2.200 | | 1.400 | X | 1.400 | | 0.800 | ****X | 0.800 | | 0.000 | X | 0.000 | +--------+-------------------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 01:52:57,960 | test | WARNING | Thread-54345 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:2Yu7WUt04vly4cMW-7-243000:VqnqEKmy4WapJB1Ap-7-359000 create failed keys from task: [u'test_collections-245'] 2021-05-18 01:52:57,961 | test | WARNING | Thread-54346 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:2Yu7WUt04vly4cMW-7-243000:Uk3CUVjZvVvlKNjgFii-7-359000 create failed keys from task: [u'test_collections-245'] 2021-05-18 01:52:57,964 | test | WARNING | Thread-54350 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:yy-7-650000:1RcYADNTEg3Lt7E-7-832000 create failed keys from task: [u'test_collections-606', u'test_collections-846', u'test_collections-711', u'test_collections-245', u'test_collections-1059', u'test_collections-796', u'test_collections-681'] 2021-05-18 01:52:57,999 | test | WARNING | Thread-54464 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:bD3pUNQLCQDk-8-89000:OjuZnwMMhH-uP8NY_t%t-8-280000 create failed keys from task: [u'test_collections-3438'] 2021-05-18 01:52:58,003 | test | WARNING | Thread-54474 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:7422hX-47-365000 update failed keys from task: [u'test_collections-859', u'test_collections-735', u'test_collections-513', u'test_collections-800', u'test_collections-864', u'test_collections-784', u'test_collections-770'] 2021-05-18 01:52:58,009 | test | WARNING | Thread-54493 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:EjcAJFKtIylxf8s66-47-365000 update failed keys from task: [u'test_collections-708', u'test_collections-729', u'test_collections-623', u'test_collections-687', u'test_collections-666', u'test_collections-915', u'test_collections-757', u'test_collections-834', u'test_collections-516', u'test_collections-657', u'test_collections-712', u'test_collections-798', u'test_collections-694'] 2021-05-18 01:52:58,015 | test | WARNING | Thread-54512 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:Y-47-376000:Oe3MhI-47-379000 update failed keys from task: [u'test_collections-819', u'test_collections-509', u'test_collections-836', u'test_collections-810', u'test_collections-556', u'test_collections-773', u'test_collections-895', u'test_collections-730', u'test_collections-893'] 2021-05-18 01:52:58,022 | test | WARNING | Thread-54539 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8xx7J4_-HC_0-8-788000:QIT-8-919000 create failed keys from task: [u'test_collections-1341', u'test_collections-1342', u'test_collections-1970'] 2021-05-18 01:52:58,022 | test | WARNING | Thread-54540 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8xx7J4_-HC_0-8-788000:SH70jsnwF5oQWiWb_0Xl-8-920000 create failed keys from task: [u'test_collections-3732'] 2021-05-18 01:52:58,028 | test | WARNING | Thread-54563 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9x75xAfHFO4Ub-47-373000:rTyRix6j-V-47-375000 delete failed keys from task: [u'test_collections-4', u'test_collections-69', u'test_collections-76', u'test_collections-98', u'test_collections-33', u'test_collections-135', u'test_collections-165', u'test_collections-152', u'test_collections-170', u'test_collections-192'] 2021-05-18 01:52:58,058 | test | WARNING | Thread-54674 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:BanyxcFj6PkTQk4WeMM-47-336000 update failed keys from task: [u'test_collections-4'] 2021-05-18 01:52:58,059 | test | WARNING | Thread-54683 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:_default:RoMY5-47-336000 delete failed keys from task: [u'test_collections-1'] 2021-05-18 01:52:58,062 | test | WARNING | Thread-54694 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:JdyfE-5-880000:2UwBsUidLk4L-6-42000 create failed keys from task: [u'test_collections-1941'] 2021-05-18 01:52:58,063 | test | WARNING | Thread-54696 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:OjhqHq3XV-6-297000:vM6Hq0Qyl2-6-629000 create failed keys from task: [u'test_collections-2620'] 2021-05-18 01:52:58,063 | test | WARNING | Thread-54700 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:vFg6Zklocb_d16Soap-5-220000:IhDmPlC48%C_-9-151000 create failed keys from task: [u'test_collections-248'] 2021-05-18 01:52:58,104 | test | WARNING | Thread-54545 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9x75xAfHFO4Ub-47-373000:foOziQFj-47-374000 update failed keys from task: [u'test_collections-827', u'test_collections-608', u'test_collections-905', u'test_collections-949', u'test_collections-548', u'test_collections-889', u'test_collections-668', u'test_collections-885', u'test_collections-840', u'test_collections-682', u'test_collections-939', u'test_collections-937', u'test_collections-955', u'test_collections-698', u'test_collections-555', u'test_collections-633'] 2021-05-18 01:52:58,102 | test | WARNING | Thread-54542 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:8ryw-3K-8-447000:xRP_0CDZ_0-8-584000 create failed keys from task: [u'test_collections-2707', u'test_collections-2706', u'test_collections-2697'] 2021-05-18 01:52:58,105 | test | WARNING | Thread-54698 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:vFg6Zklocb_d16Soap-5-220000:1Z4y-qckzt0SvAAZBS-5-470000 create failed keys from task: [u'test_collections-1415', u'test_collections-1244', u'test_collections-1585'] 2021-05-18 01:52:58,141 | test | WARNING | Thread-54697 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:OjhqHq3XV-6-297000:7QFrBLiMt3ZudDtWLz-6-628000 create failed keys from task: [u'test_collections-1941'] 2021-05-18 01:52:58,174 | test | WARNING | Thread-54568 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9x75xAfHFO4Ub-47-373000:7YmIAiuKc8pBlAgfP-47-375000 update failed keys from task: [u'test_collections-704', u'test_collections-867', u'test_collections-978', u'test_collections-822', u'test_collections-896'] 2021-05-18 01:52:58,174 | test | WARNING | Thread-54574 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9x75xAfHFO4Ub-47-373000:o-47-376000 update failed keys from task: [u'test_collections-934'] 2021-05-18 01:52:58,196 | test | WARNING | Thread-54575 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:9x75xAfHFO4Ub-47-373000:BxFBjUuOorRVL9N0kpE-47-375000 delete failed keys from task: [u'test_collections-44', u'test_collections-117', u'test_collections-84', u'test_collections-133', u'test_collections-166', u'test_collections-101', u'test_collections-176', u'test_collections-152', u'test_collections-192', u'test_collections-7'] 2021-05-18 01:52:58,227 | test | WARNING | Thread-54590 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:bwlRB5Osu-47-367000 update failed keys from task: [u'test_collections-827', u'test_collections-608', u'test_collections-905', u'test_collections-949', u'test_collections-889', u'test_collections-626', u'test_collections-885', u'test_collections-644', u'test_collections-840', u'test_collections-939', u'test_collections-937', u'test_collections-955', u'test_collections-555', u'test_collections-633'] 2021-05-18 01:52:58,237 | test | WARNING | Thread-54628 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:collection2 delete failed keys from task: [u'test_collections-25', u'test_collections-103', u'test_collections-92', u'test_collections-144', u'test_collections-166', u'test_collections-101', u'test_collections-123', u'test_collections-7'] 2021-05-18 01:52:58,253 | test | WARNING | Thread-54594 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:collection_1 update failed keys from task: [u'test_collections-648', u'test_collections-522', u'test_collections-567', u'test_collections-618', u'test_collections-614', u'test_collections-578', u'test_collections-531', u'test_collections-651', u'test_collections-574', u'test_collections-593', u'test_collections-692'] 2021-05-18 01:52:58,256 | test | WARNING | Thread-54645 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:6-9-152000 create failed keys from task: [u'test_collections-2977'] 2021-05-18 01:52:58,266 | test | WARNING | Thread-54648 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:vs53nd379nT-47-373000 update failed keys from task: [u'test_collections-625', u'test_collections-769', u'test_collections-821', u'test_collections-545', u'test_collections-784', u'test_collections-818', u'test_collections-859', u'test_collections-515', u'test_collections-559', u'test_collections-735', u'test_collections-599', u'test_collections-872', u'test_collections-794', u'test_collections-770'] 2021-05-18 01:52:58,313 | test | WARNING | Thread-54654 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:7Q_vGMN7VVgAY_-47-373000 update failed keys from task: [u'test_collections-708', u'test_collections-849', u'test_collections-528', u'test_collections-915', u'test_collections-757', u'test_collections-834', u'test_collections-516', u'test_collections-956', u'test_collections-712', u'test_collections-798', u'test_collections-694'] 2021-05-18 01:52:58,336 | test | WARNING | Thread-54600 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:xuJ-47-367000 update failed keys from task: [u'test_collections-607', u'test_collections-629', u'test_collections-648', u'test_collections-522', u'test_collections-531', u'test_collections-642', u'test_collections-574', u'test_collections-593', u'test_collections-692'] 2021-05-18 01:52:58,612 | test | WARNING | Thread-54635 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope2:KZMNUjXe7B92z-47-373000 delete failed keys from task: [u'test_collections-98', u'test_collections-206', u'test_collections-243', u'test_collections-165', u'test_collections-286', u'test_collections-215', u'test_collections-135', u'test_collections-371', u'test_collections-250', u'test_collections-293', u'test_collections-170', u'test_collections-76', u'test_collections-33', u'test_collections-346', u'test_collections-347', u'test_collections-303', u'test_collections-385', u'test_collections-4', u'test_collections-69', u'test_collections-316', u'test_collections-353', u'test_collections-396', u'test_collections-152', u'test_collections-192'] 2021-05-18 01:53:08,365 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 01:53:59,490 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 01:53:59,490 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 01:54:17,558 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 01:54:17,559 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 01:54:36,688 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 01:54:36,688 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 01:56:00,444 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1270000935 seconds to complete 2021-05-18 01:56:00,460 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2983.573s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 23 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_23 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_24 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=2,nodes_out=1,override_spec_params=durability;replicas,durability=PERSIST_TO_MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'PERSIST_TO_MAJORITY', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_24', 'nodes_init': '4', 'GROUP': 'rebalance_with_collection_crud_durability_PERSIST_TO_MAJORITY', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 24, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 01:56:09,852 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #24 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 01:56:10,542 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #24 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 01:56:11,016 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x6e79bb type=client open_count=1 channel=[id: 0xf7c40ed7, 0.0.0.0/0.0.0.0:35896] timeout=300.0>) 2021-05-18 01:56:31,848 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x6e79bc type=client open_count=1 channel=[id: 0xfc6115be, 0.0.0.0/0.0.0.0:58939] timeout=300.0>) 2021-05-18 01:56:33,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x6e79bd type=client open_count=1 channel=[id: 0x159d8969, 0.0.0.0/0.0.0.0:47252] timeout=300.0>) 2021-05-18 01:56:35,884 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x6e79be type=client open_count=1 channel=[id: 0x2120c44a, 0.0.0.0/0.0.0.0:55928] timeout=300.0>) 2021-05-18 01:57:00,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x6e79bf type=client open_count=1 channel=[id: 0x0b876c85, 0.0.0.0/0.0.0.0:55170] timeout=300.0>) 2021-05-18 01:57:02,316 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x6e79c0 type=client open_count=1 channel=[id: 0x0ee5061a, 0.0.0.0/0.0.0.0:57504] timeout=300.0>) 2021-05-18 01:57:04,334 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x6e79c1 type=client open_count=1 channel=[id: 0x4df03e70, 0.0.0.0/0.0.0.0:39578] timeout=300.0>) 2021-05-18 01:57:28,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x6e79c2 type=client open_count=1 channel=[id: 0x2da6ef87, 0.0.0.0/0.0.0.0:57703] timeout=300.0>) 2021-05-18 01:57:30,440 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x6e79c3 type=client open_count=1 channel=[id: 0x42dd771b, 0.0.0.0/0.0.0.0:34144] timeout=300.0>) 2021-05-18 01:57:32,457 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x6e79c4 type=client open_count=1 channel=[id: 0xb1122036, 0.0.0.0/0.0.0.0:46853] timeout=300.0>) 2021-05-18 01:57:56,634 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x6e79c5 type=client open_count=1 channel=[id: 0xb9b6b3f9, 0.0.0.0/0.0.0.0:58129] timeout=300.0>) 2021-05-18 01:57:58,653 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x6e79c6 type=client open_count=1 channel=[id: 0x3b92a129, 0.0.0.0/0.0.0.0:42641] timeout=300.0>) 2021-05-18 01:58:00,670 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x6e79c7 type=client open_count=1 channel=[id: 0xbc2e250e, 0.0.0.0/0.0.0.0:54679] timeout=300.0>) 2021-05-18 01:58:02,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x6e79c8 type=client open_count=1 channel=[id: 0x2ea18651, 0.0.0.0/0.0.0.0:34112] timeout=300.0>) 2021-05-18 01:58:26,032 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x6e79c9 type=client open_count=1 channel=[id: 0x232ef242, 0.0.0.0/0.0.0.0:34578] timeout=300.0>) 2021-05-18 01:58:28,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x6e79ca type=client open_count=1 channel=[id: 0x8c172f49, 0.0.0.0/0.0.0.0:35581] timeout=300.0>) 2021-05-18 01:58:30,068 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 01:58:47,198 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 01:59:00,971 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 01:59:00,994 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #24 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 01:59:01,035 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #24 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 01:59:10,786 | test | INFO | pool-26-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.53159114015 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 01:59:10,832 | test | INFO | pool-26-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 01:59:20,914 | test | INFO | pool-26-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 01:59:20,943 | test | INFO | pool-26-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1570000648 sec 2021-05-18 01:59:21,230 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 02:00:17,372 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 02:00:17,673 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 838860800 | 642128928 | 96197907 | | bucket2 | couchbase | 2 | none | 0 | 0 | 838860800 | 579607632 | 114787610 | | default | couchbase | 2 | none | 0 | 0 | 8388608000 | 662072352 | 82240090 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-18 02:00:17,673 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 02:01:00,013 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 02:01:00,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 02:23:18,217 | test | INFO | pool-26-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***........................................................*****.......................*.....................****X | 768.261 | | 691.516 | .........................................................................................X | 691.516 | | 658.816 | ....................................................................................****X | 658.816 | | 584.262 | ...........................................................................X | 584.262 | | 517.450 | ..................................................................******..*****.............*....****X | 662.583 | | 564.829 | ........................................................................*****X | 564.829 | | 505.676 | ................................................................****X | 505.676 | | 481.444 | .............................................................X | 481.444 | | 473.541 | ............................................................*****..*....****X | 516.024 | | 418.562 | .....................................................****X | 418.562 | | 400.853 | ...................................................X | 400.853 | | 391.110 | .................................................****X | 391.110 | | 388.548 | .................................................X | 388.548 | | 385.358 | .................................................*****.*.*****...*****X | 429.208 | | 368.356 | ..............................................*****X | 368.356 | | 365.256 | ..............................................******.......*****X | 419.239 | | 371.262 | ...............................................*****.X | 382.525 | | 365.537 | ..............................................*****..........*****X | 445.178 | | 420.088 | .....................................................*****X | 420.088 | | 401.652 | ...................................................*****.*...***X | 433.137 | | 425.513 | ......................................................X | 425.513 | | 408.750 | ....................................................X | 408.750 | | 404.050 | ...................................................***X | 404.050 | | 385.700 | .................................................******X | 386.819 | | 375.675 | ...............................................****X | 375.675 | | 325.943 | .........................................******...*****...*..****X | 386.043 | | 361.717 | ..............................................*****X | 363.625 | | 354.425 | .............................................****X | 354.425 | | 326.099 | .........................................*****....*..*****.X | 380.828 | | 375.391 | ...............................................****X | 375.391 | | 358.428 | .............................................*****....*****X | 390.164 | | 318.078 | ........................................******..........*****X | 396.011 | | 331.690 | ..........................................X | 331.690 | | 307.930 | .......................................***X | 307.930 | | 253.043 | ...............................X | 253.043 | | 234.643 | .............................****X | 234.643 | | 231.243 | .............................X | 231.243 | | 210.301 | ..........................*****........******.............****X | 370.481 | | 350.825 | ............................................X | 350.825 | | 343.525 | ...........................................****X | 343.525 | | 336.008 | ..........................................X | 336.008 | | 325.243 | .........................................*****...*****..*.....****X | 405.001 | | 341.083 | ...........................................****X | 341.083 | | 318.492 | ........................................******.*****...******...*****X | 374.888 | | 348.326 | ............................................*****X | 348.326 | | 322.282 | ........................................******............***********X | 412.388 | | 372.166 | ...............................................***********.X | 377.685 | | 369.090 | ...............................................*****..*****X | 386.651 | | 325.775 | .........................................******.....*****X | 368.226 | | 353.735 | .............................................X | 353.735 | | 337.591 | ..........................................*****...*****X | 359.540 | | 331.216 | ..........................................X | 331.216 | | 319.865 | ........................................*****...*.*****......****X | 395.211 | | 385.692 | .................................................X | 385.692 | | 329.235 | .........................................*****.*****X | 336.183 | | 332.828 | ..........................................******.....*****X | 375.118 | | 356.120 | .............................................*****X | 356.120 | | 270.308 | ..................................******...*..........******.*****...****X | 404.239 | | 361.346 | ..............................................*****X | 361.346 | | 329.431 | .........................................X | 329.431 | | 329.048 | .........................................***X | 329.048 | | 320.201 | ........................................X | 320.201 | | 285.437 | ....................................****X | 285.437 | | 279.918 | ...................................X | 279.918 | | 255.013 | ................................*****..........******......******....*****X | 414.774 | | 348.065 | ............................................****X | 348.065 | | 314.600 | .......................................******..*...*****...******.***********X | 383.496 | | 353.779 | .............................................*****X | 353.779 | | 337.580 | ..........................................******............****X | 424.825 | | 396.050 | ..................................................X | 396.050 | | 358.819 | .............................................****X | 358.819 | | 340.867 | ...........................................*****X | 344.360 | | 338.817 | ...........................................X | 338.817 | | 334.017 | ..........................................****X | 334.017 | | 283.310 | ...................................******.........*****X | 352.594 | | 346.355 | ............................................************...X | 368.786 | | 359.208 | .............................................***X | 359.208 | | 327.475 | .........................................*.*****.X | 338.694 | | 329.436 | .........................................*****.******....*..***X | 379.277 | | 363.877 | ..............................................X | 363.877 | | 310.033 | .......................................*****...***********.X | 340.464 | | 339.126 | ...........................................*****.....*****X | 377.997 | | 373.186 | ...............................................*****.*****X | 381.995 | | 372.237 | ...............................................X | 372.237 | | 362.350 | ..............................................****X | 362.350 | | 329.629 | .........................................X | 329.629 | | 316.003 | ........................................*****.....*****X | 360.421 | | 353.563 | .............................................******....*****X | 386.840 | | 287.018 | ....................................*****.*...*****..*...****X | 357.748 | | 351.359 | ............................................X | 351.359 | | 330.450 | ..........................................****X | 330.450 | | 329.634 | .........................................X | 329.634 | | 324.894 | .........................................****X | 324.894 | | 312.956 | .......................................******.....*****X | 346.309 | | 333.277 | ..........................................***********..*****X | 347.884 | | 343.092 | ...........................................*..*****....*.....*****.*.*****.*****.X | 456.014 | | 440.458 | ........................................................*****..*****X | 458.049 | | 279.389 | ...................................X | 279.389 | +---------+--------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:23:19,451 | test | INFO | pool-26-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***..........****X | 4.500 | | 2.400 | ....X | 2.400 | | 0.000 | *********************************************************************************************************************************************...******....*****X | 3.000 | | 0.000 | **********************************************...****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | *****.......................*...........................****X | 20.998 | | 0.000 | ******...........................................................................*****.............*............****X | 41.306 | | 34.000 | .................................................................................*****X | 34.000 | | 6.000 | .............*****X | 6.000 | | 0.000 | ******...*****X | 1.500 | | 0.000 | ******...****X | 1.500 | | 0.000 | ***********.....*..****X | 3.000 | | 0.000 | *****************.......*****X | 2.997 | | 0.000 | ******************.*..****X | 1.499 | | 0.000 | *****************...*....****X | 2.998 | | 0.000 | ******...******.....*..*****...*****X | 5.597 | | 3.397 | .......*****X | 3.397 | | 1.499 | ..*****X | 1.499 | | 1.499 | ..*****X | 1.499 | | 0.000 | ******...****X | 1.499 | | 0.399 | X | 0.399 | | 0.000 | *****.......******..........*..............................****X | 19.486 | | 1.499 | ..*****X | 1.499 | | 0.400 | X | 0.400 | | 0.000 | ***********.*..*****......................****X | 10.595 | | 8.396 | ...................X | 8.396 | | 0.000 | *****..................*..................................................****X | 28.479 | | 0.000 | *****************..............******...............*****X | 11.996 | | 1.500 | ..*****X | 1.500 | | 0.000 | ************************************...****X | 1.500 | | 1.100 | .X | 1.100 | | 0.000 | ***********................*....................****X | 14.995 | | 0.000 | ******..........*****X | 4.499 | | 0.000 | ***********************************...****X | 1.498 | | 0.000 | *****************************************..................*****X | 7.496 | | 0.000 | ************...******....**********X | 3.002 | | 0.000 | ******...*****X | 1.501 | | 0.000 | **********************************************************************X | 0.000 | +--------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 02:23:19,638 | test | INFO | pool-26-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *********************************************************************************************************************************************************************************************************************************************...*.........*****.......................*.................................................................****X | 54.999 | | 37.401 | ...................................................................*****X | 37.401 | | 0.000 | *****........................*****X | 13.501 | | 9.300 | ...............X | 9.300 | | 4.499 | .......*****.........****X | 9.500 | | 1.000 | ******.**********X | 1.500 | | 0.000 | *****..*****X | 1.499 | | 0.000 | ******.....*****X | 2.997 | | 2.997 | ....*****X | 2.997 | | 0.000 | ************..*****X | 1.499 | | 0.000 | ******************************....****X | 2.599 | | 2.599 | ...*..****X | 3.397 | | 3.396 | .....X | 3.396 | | 3.098 | ....*****X | 3.098 | | 2.898 | ....****X | 2.898 | | 1.898 | ..X | 1.898 | | 1.499 | .*****..*....****X | 4.496 | | 0.000 | ************..........******.............******.......*****X | 16.887 | | 0.000 | ***********************.................*.................****X | 19.091 | | 13.898 | ........................******......****X | 17.586 | | 0.000 | ******.......*****X | 4.298 | | 1.699 | ..*****X | 1.699 | | 0.000 | ******..******......****X | 4.499 | | 0.000 | ******************************.....*****X | 2.998 | | 1.500 | .*****X | 1.500 | | 0.000 | ************..........*****X | 5.998 | | 0.000 | ******..****X | 1.500 | | 0.400 | X | 0.400 | | 0.000 | **********************************.*....****X | 2.997 | | 0.800 | X | 0.800 | | 0.000 | *****************************..****X | 1.499 | | 1.099 | X | 1.099 | | 0.000 | *****..*...****X | 2.999 | | 1.399 | .X | 1.399 | | 0.000 | ***********....*..........*****X | 7.796 | | 0.800 | *****..*.....****X | 4.503 | | 0.000 | ******..******...*****....*****X | 5.103 | | 0.900 | *****X | 0.900 | | 0.000 | ****************************************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 02:23:19,887 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 72.8654522425 | 6437847040 | 4129660928 | 2566840320 / 7511588864 | 125691 / 250765 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 76.875 | 6437847040 | 4158115840 | 2538684416 / 7511588864 | 125951 / 255702 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 78.5597435983 | 6437847040 | 3726815232 | 3091656704 / 9530753024 | 126767 / 248129 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 89.2981549692 | 6437847040 | 4168945664 | 2566795264 / 7511588864 | 126754 / 255679 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 02:23:30,773 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 02:24:41,082 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 838860800 | 158847680 | 269846310 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 838860800 | 158156704 | 278085188 | | default | couchbase | 2 | none | 0 | 500000 | 8388608000 | 438207376 | 457592078 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 02:24:41,085 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #24 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 02:24:49,167 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-18 02:24:49,168 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-18 02:24:52,753 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 02:24:52,980 | test | INFO | pool-26-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 16.7222185188 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 10.8583333333 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 14.665 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 11.1448142531 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 02:24:53,073 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 02:24:58,288 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 02:25:02,655 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 02:25:03,391 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.27 2021-05-18 02:25:08,510 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 2.15 2021-05-18 02:25:13,621 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 4.5 2021-05-18 02:25:16,092 | test | INFO | pool-26-thread-14 | [table_view:display:72] Keys failed in bucket1:_default:Ca2_et2cQj-21-300000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:16,092 | test | INFO | pool-26-thread-29 | [table_view:display:72] Keys failed in bucket1:_default:wxO79BO9JEyMmQT86Qy-21-301000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:16,092 | test | INFO | pool-26-thread-7 | [table_view:display:72] Keys failed in bucket1:scope2:EaIrOcma7IYrSSY-21-307000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:16,094 | test | INFO | pool-26-thread-1 | [table_view:display:72] Keys failed in bucket1:_default:SYpeIpCNOip7dZ_YS6-21-300000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:16,094 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket1:scope2:sjN5pAxS-21-308000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:16,094 | test | INFO | pool-26-thread-12 | [table_view:display:72] Keys failed in bucket1:scope2:S-21-307000 +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | +--------------------+-----------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:18,736 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.08 2021-05-18 02:25:23,872 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 9.92 2021-05-18 02:25:28,996 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 13.26 2021-05-18 02:25:34,112 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 17.41 2021-05-18 02:25:39,219 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 20.83 2021-05-18 02:25:39,316 | test | INFO | pool-26-thread-1 | [table_view:display:72] Keys failed in bucket1:bNJLQpKKvZjC-59-738000:hJQxAu-0-970000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:39,318 | test | INFO | pool-26-thread-1 | [table_view:display:72] Keys failed in bucket1:bNJLQpKKvZjC-59-738000:hJQxAu-0-970000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-134 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:25:44,369 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 24.41 2021-05-18 02:25:49,520 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 27.89 2021-05-18 02:25:54,617 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.09 2021-05-18 02:25:59,697 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 02:26:04,779 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 02:26:09,857 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 02:26:14,980 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 35.47 2021-05-18 02:26:20,135 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 38.39 2021-05-18 02:26:25,233 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.22 2021-05-18 02:26:30,335 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-18 02:26:35,447 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 46.03 2021-05-18 02:26:40,565 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 48.49 2021-05-18 02:26:45,658 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 51.92 2021-05-18 02:26:50,684 | test | INFO | pool-26-thread-18 | [table_view:display:72] Keys failed in bucket2:dMj%jhQo_6xfZhVrV-58-76000:q5G7ii-ZPX-58-346000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:26:50,686 | test | INFO | pool-26-thread-18 | [table_view:display:72] Keys failed in bucket2:dMj%jhQo_6xfZhVrV-58-76000:q5G7ii-ZPX-58-346000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1172 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:26:50,746 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 54.94 2021-05-18 02:26:55,855 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 57.58 2021-05-18 02:27:00,964 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 60.91 2021-05-18 02:27:06,085 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 63.39 2021-05-18 02:27:06,470 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket2:pmFOP9E-57-582000:IqO4CgzSpmoZax-57-826000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:06,470 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket2:pmFOP9E-57-582000:IqO4CgzSpmoZax-57-826000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1547 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:11,193 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.99 2021-05-18 02:27:16,285 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 02:27:16,951 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket2:pmFOP9E-57-582000:IqO4CgzSpmoZax-57-826000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:16,953 | test | INFO | pool-26-thread-3 | [table_view:display:72] Keys failed in bucket2:pmFOP9E-57-582000:IqO4CgzSpmoZax-57-826000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1764 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:20,687 | test | INFO | pool-26-thread-10 | [table_view:display:72] Keys failed in bucket2:dMj%jhQo_6xfZhVrV-58-76000:S%K_45Mz79jj-58-346000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:20,687 | test | INFO | pool-26-thread-10 | [table_view:display:72] Keys failed in bucket2:dMj%jhQo_6xfZhVrV-58-76000:S%K_45Mz79jj-58-346000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-948 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:27:21,364 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 02:27:27,354 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.71 2021-05-18 02:27:32,441 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 69.01 2021-05-18 02:27:37,558 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.79 2021-05-18 02:27:42,694 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 72.79 2021-05-18 02:27:47,806 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 74.32 2021-05-18 02:27:52,914 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 76.11 2021-05-18 02:27:58,036 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 77.84 2021-05-18 02:27:58,309 | test | INFO | pool-26-thread-4 | [table_view:display:72] Keys failed in default:scope1:uKR8cX7jMS%Rh0A-21-319000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-646 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:28:02,707 | test | INFO | pool-26-thread-17 | [table_view:display:72] Keys failed in default:_default:5CqGMMx5J3n12k-21-314000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-487 | {'error': 'com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:28:03,117 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 79.32 2021-05-18 02:28:08,200 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 81.28 2021-05-18 02:28:13,283 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 82.73 2021-05-18 02:28:18,375 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 84.76 2021-05-18 02:28:23,482 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 86.6 2021-05-18 02:28:28,598 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 88.15 2021-05-18 02:28:33,717 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 89.83 2021-05-18 02:28:38,822 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 91.64 2021-05-18 02:28:43,983 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 93.58 2021-05-18 02:28:49,066 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 95.49 2021-05-18 02:28:54,164 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 97.55 2021-05-18 02:28:59,263 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 02:29:04,357 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 02:29:09,469 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 02:29:16,848 | test | INFO | pool-26-thread-1 | [table_view:display:72] Keys failed in default:--eHxBSkvecQfp-21-325000:9BOnYlfUE-7dx-21-326000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-529 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:29:19,717 | test | INFO | pool-26-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 02:29:19,759 | test | INFO | pool-26-thread-23 | [table_view:display:72] Keys failed in default:--eHxBSkvecQfp-21-325000:9BOnYlfUE-7dx-21-326000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2614 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:29:19,759 | test | INFO | pool-26-thread-23 | [table_view:display:72] Keys failed in default:--eHxBSkvecQfp-21-325000:9BOnYlfUE-7dx-21-326000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2614 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-18 02:29:19,828 | test | INFO | pool-26-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 266.848999977 sec 2021-05-18 02:32:30,783 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,783 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,786 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,786 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,786 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,788 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 3 to 5 2021-05-18 02:32:30,789 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 2 docs from 0 to 5000 2021-05-18 02:32:30,789 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 02:32:54,316 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 02:32:54,321 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 0 to 5000 2021-05-18 02:32:54,328 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 0 to 500 2021-05-18 02:32:54,332 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-18 02:32:54,336 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 500 to 980 2021-05-18 02:32:54,338 | test | ERROR | pool-26-thread-27 | [task:call:3809] Failed to load 1 docs from 2500 to 2900 2021-05-18 02:40:55,265 | test | INFO | pool-26-thread-16 | [table_view:display:72] Ops trend for bucket 'default' +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****............................................***X | 322.165 | | 245.478 | ................................****..........***X | 313.856 | | 237.883 | ...............................X | 237.883 | | 227.974 | ..............................***......*..***X | 286.620 | | 271.590 | ....................................*.........***X | 338.567 | | 338.567 | .............................................X | 338.567 | | 319.170 | ..........................................***X | 319.170 | | 278.698 | .....................................X | 278.698 | | 171.224 | ......................****..................****..........***X | 371.903 | | 271.451 | ....................................X | 271.451 | | 238.580 | ...............................****..........*****........*..X | 380.237 | | 380.237 | ...................................................**X | 380.237 | | 373.345 | ..................................................*..*****.***X | 393.681 | | 176.157 | .......................*****...........***X | 254.182 | | 144.653 | ..................X | 144.653 | | 111.626 | ..............****..................*.....***X | 278.324 | | 198.350 | ..........................***X | 198.350 | | 171.262 | ......................X | 171.262 | | 145.623 | ...................***X | 145.623 | | 117.134 | ...............X | 117.134 | | 87.385 | ...........****.....*****..................*......X | 303.476 | | 301.576 | ........................................***X | 301.576 | | 187.453 | ........................*****....*.*****....*.****....*....****X | 314.959 | | 289.558 | ......................................X | 289.558 | | 271.039 | ....................................****X | 271.039 | | 251.133 | .................................X | 251.133 | | 247.743 | .................................****X | 247.743 | | 188.140 | ........................X | 188.140 | | 166.919 | .....................****X | 166.919 | | 149.617 | ...................X | 149.617 | | 148.712 | ...................*****......*.*****............****X | 283.386 | | 255.899 | ..................................X | 255.899 | | 192.410 | .........................X | 192.410 | | 160.515 | .....................****..........*....****X | 261.406 | | 199.285 | ..........................*****X | 199.285 | | 192.113 | .........................****X | 192.113 | | 184.609 | ........................*..****X | 196.514 | | 168.583 | ......................******..******...*****........................................................................X | 725.765 | | 725.765 | ..................................................................................................****X | 725.765 | | 498.425 | ...................................................................******.****X | 503.950 | | 448.501 | ............................................................*X | 448.501 | | 448.501 | ............................................................*X | 448.501 | | 448.501 | ............................................................**.....................................***X | 717.790 | | 717.790 | .................................................................................................X | 717.790 | | 534.000 | ........................................................................*****X | 534.000 | | 492.229 | ..................................................................*X | 492.229 | | 492.229 | ..................................................................****........*********X | 550.700 | | 550.700 | ..........................................................................X | 550.700 | | 515.112 | .....................................................................****X | 515.112 | | 460.500 | ..............................................................X | 460.500 | | 273.986 | ....................................****X | 273.986 | | 246.219 | ................................*****X | 246.219 | | 225.201 | ..............................*****X | 225.201 | | 209.095 | ...........................*****..*****X | 224.616 | | 199.790 | ..........................******............*****...*.........*****.......*****X | 421.348 | | 322.633 | ...........................................*****.........*****X | 385.498 | | 340.904 | .............................................X | 340.904 | | 334.885 | .............................................****..*...*****.X | 378.350 | | 328.772 | ............................................**X | 328.772 | | 328.772 | ............................................**...****X | 354.655 | | 313.572 | ..........................................******.....*......****.*****X | 402.539 | | 316.616 | ..........................................******.....****X | 353.508 | | 315.785 | ..........................................******................*****X | 433.500 | | 349.891 | ...............................................*****.*.***X | 364.082 | | 358.004 | ................................................X | 358.004 | | 320.180 | ...........................................****X | 320.180 | | 318.332 | ..........................................X | 318.332 | | 310.576 | .........................................******..*****.....******...**********X | 384.110 | | 372.572 | ..................................................*****X | 372.572 | | 360.382 | ................................................****X | 360.382 | | 349.248 | ...............................................******...******.....****X | 407.670 | | 393.524 | .....................................................X | 393.524 | | 382.253 | ...................................................*.****.....*****X | 422.952 | | 411.300 | .......................................................*****..*..*****....******...*****...*........***X | 571.042 | +---------+-----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:40:55,891 | test | INFO | pool-26-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****...***X | 7.201 | | 5.403 | .*...........****............****.............***X | 77.848 | | 66.510 | ................................X | 66.510 | | 58.628 | ............................*****...........................***X | 113.534 | | 94.810 | ..............................................X | 94.810 | | 76.934 | .....................................****.*..**X | 83.593 | | 81.520 | .......................................*.....****.................................***X | 157.337 | | 143.170 | ......................................................................X | 143.170 | | 141.400 | .....................................................................***X | 141.400 | | 120.398 | ..........................................................X | 120.398 | | 114.546 | .......................................................****...................................*........****X | 201.482 | | 188.000 | ............................................................................................***X | 188.000 | | 148.820 | ........................................................................X | 148.820 | | 115.998 | ........................................................***X | 115.998 | | 109.748 | .....................................................****X | 109.748 | | 95.753 | ..............................................***X | 95.753 | | 78.021 | .....................................X | 78.021 | | 71.316 | ..................................****...................*......***X | 121.876 | | 100.113 | ................................................****...........*........***X | 138.641 | | 132.396 | ................................................................X | 132.396 | | 114.172 | .......................................................****...****X | 120.587 | | 72.941 | ...................................X | 72.941 | | 57.528 | ...........................*****......*****.........*.****X | 90.025 | | 85.801 | .........................................X | 85.801 | | 84.500 | ........................................*****.******.*..*****..........*****X | 112.219 | | 82.109 | .......................................****X | 82.109 | | 78.910 | ......................................X | 78.910 | | 60.904 | .............................****X | 60.904 | | 56.906 | ...........................X | 56.906 | | 47.501 | ......................****X | 47.501 | | 44.899 | .....................X | 44.899 | | 43.298 | ....................****X | 43.298 | | 40.504 | ...................*****..*..*..***X | 53.402 | | 45.296 | .....................*****X | 45.296 | | 42.103 | ...................*.**********X | 43.903 | | 42.695 | ....................*****......*****X | 55.197 | | 21.202 | .........*****X | 21.202 | | 2.800 | *****X | 2.800 | | 1.100 | ****X | 1.100 | | 0.800 | ******.*****X | 3.500 | | 1.300 | *****.*******.****X | 4.299 | | 3.101 | ****X | 3.101 | | 1.200 | **********X | 1.600 | | 1.500 | *****X | 1.700 | | 1.400 | ****X | 1.400 | | 0.400 | X | 0.400 | | 0.200 | ****X | 0.200 | | 0.100 | X | 0.100 | | 0.000 | ****************X | 1.900 | | 0.400 | ****X | 0.400 | | 0.300 | ******.*****X | 2.500 | | 1.300 | ****X | 1.300 | | 0.400 | ******.*****X | 2.100 | | 0.600 | ************.****X | 2.802 | | 2.801 | *****X | 3.001 | | 0.600 | ******.*****X | 3.300 | | 0.800 | *****X | 0.800 | | 0.700 | **********X | 1.800 | | 0.900 | *****X | 0.900 | | 0.200 | ******.****X | 2.198 | | 1.501 | ****X | 1.501 | | 0.000 | **********X | 0.700 | | 0.600 | ***********.*****X | 2.300 | | 0.900 | ***********X | 1.600 | | 1.200 | X | 1.200 | | 0.700 | *********X | 0.900 | | 0.400 | X | 0.400 | | 0.000 | ***********.*****X | 3.100 | | 2.400 | ****X | 2.400 | | 1.500 | X | 1.500 | | 0.700 | *****.*****X | 2.900 | | 2.500 | X | 2.500 | | 1.999 | ***X | 1.999 | | 1.501 | X | 1.501 | | 1.301 | ****X | 1.301 | +---------+--------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:40:56,854 | test | INFO | pool-26-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *****..****...........X | 13.900 | | 13.900 | ............**X | 13.900 | | 13.499 | ...........****....................*......***X | 40.911 | | 33.127 | ..............................*****...............*****..****X | 51.260 | | 32.540 | .............................****..*****.....*...**X | 42.313 | | 41.809 | ......................................X | 41.809 | | 40.088 | .....................................****...........***X | 52.432 | | 48.721 | .............................................*......**X | 54.722 | | 52.323 | ................................................**......................*.......**X | 82.202 | | 76.708 | .......................................................................X | 76.708 | | 67.601 | ...............................................................****X | 68.190 | | 53.571 | .................................................****.................****X | 71.041 | | 66.413 | ..............................................................***X | 66.413 | | 59.808 | .......................................................*..****...............*....*X | 81.115 | | 81.115 | ............................................................................*X | 81.115 | | 76.430 | .......................................................................X | 76.430 | | 56.118 | ....................................................****.......................***X | 80.384 | | 80.384 | ...........................................................................X | 80.384 | | 52.193 | ................................................*****.......................****X | 75.836 | | 75.386 | ......................................................................*.*****..............*.****...X | 94.902 | | 94.099 | ........................................................................................*****.....*..*****.*...****X | 105.116 | | 99.613 | .............................................................................................X | 99.613 | | 98.918 | .............................................................................................*****....*.****X | 104.813 | | 91.912 | ......................................................................................X | 91.912 | | 85.906 | ................................................................................****X | 85.906 | | 54.002 | ..................................................*****..........*.....****X | 70.197 | | 67.507 | ...............................................................******..*****X | 70.401 | | 48.596 | .............................................*****....******......*****X | 59.704 | | 56.995 | .....................................................*****X | 56.995 | | 55.197 | ...................................................*****X | 55.197 | | 48.205 | ............................................****X | 48.205 | | 3.500 | ..*****X | 3.500 | | 2.000 | ******.******.***********.*****X | 4.700 | | 3.399 | ..****X | 3.399 | | 3.201 | ..X | 3.201 | | 3.101 | .****X | 3.101 | | 2.700 | .*****....*.............*****..............*..........................................***X | 79.396 | | 76.896 | ........................................................................X | 76.896 | | 72.606 | ....................................................................****X | 72.606 | | 70.300 | .................................................................******.******.**********X | 72.296 | | 50.398 | ..............................................*****X | 50.398 | | 0.900 | ******.**********X | 2.100 | | 1.100 | ******.*****X | 2.200 | | 2.101 | ****X | 2.101 | | 1.800 | ******X | 1.998 | | 1.601 | ******.****X | 2.601 | | 2.101 | X | 2.101 | | 0.600 | ****.*..****X | 4.000 | | 3.600 | ..*****X | 3.600 | | 3.300 | ..******.****X | 4.400 | | 2.300 | .*****X | 2.300 | | 1.400 | *******.*****...****X | 5.703 | | 1.400 | *****X | 1.400 | | 0.400 | ***********.*****.*****X | 2.200 | | 0.600 | ******.******.****X | 2.600 | | 1.600 | X | 1.600 | | 1.000 | *****.******.*****X | 2.700 | | 2.501 | .*****.*****X | 3.800 | | 1.300 | ******...****X | 5.100 | | 4.200 | ..X | 4.200 | | 3.198 | ..****X | 3.198 | | 1.902 | *****X | 1.902 | +--------+----------------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:40:56,904 | test | WARNING | Thread-57212 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:_default:5CqGMMx5J3n12k-21-314000 delete failed keys from task: [u'test_collections-487'] 2021-05-18 02:40:56,908 | test | WARNING | Thread-57238 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:--eHxBSkvecQfp-21-325000:9BOnYlfUE-7dx-21-326000 update failed keys from task: [u'test_collections-529'] 2021-05-18 02:40:56,917 | test | WARNING | Thread-57238 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:--eHxBSkvecQfp-21-325000:9BOnYlfUE-7dx-21-326000 create failed keys from task: [u'test_collections-2614'] 2021-05-18 02:40:56,923 | test | WARNING | Thread-57296 | [bucket_ready_functions:validate_crud_task_per_collection:501] default:scope1:uKR8cX7jMS%Rh0A-21-319000 update failed keys from task: [u'test_collections-646'] 2021-05-18 02:40:56,937 | test | WARNING | Thread-57360 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:SYpeIpCNOip7dZ_YS6-21-300000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:40:56,940 | test | WARNING | Thread-57368 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:wxO79BO9JEyMmQT86Qy-21-301000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:40:56,944 | test | WARNING | Thread-57351 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:Ca2_et2cQj-21-300000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:40:56,957 | test | WARNING | Thread-57416 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:bNJLQpKKvZjC-59-738000:hJQxAu-0-970000 create failed keys from task: [u'test_collections-134'] 2021-05-18 02:40:56,996 | test | WARNING | Thread-57562 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:dMj%jhQo_6xfZhVrV-58-76000:q5G7ii-ZPX-58-346000 create failed keys from task: [u'test_collections-1172'] 2021-05-18 02:40:56,996 | test | WARNING | Thread-57563 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:dMj%jhQo_6xfZhVrV-58-76000:S%K_45Mz79jj-58-346000 create failed keys from task: [u'test_collections-948'] 2021-05-18 02:40:57,023 | test | WARNING | Thread-57642 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket2:pmFOP9E-57-582000:IqO4CgzSpmoZax-57-826000 create failed keys from task: [u'test_collections-1764', u'test_collections-1547'] 2021-05-18 02:40:57,033 | test | WARNING | Thread-57465 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:sjN5pAxS-21-308000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:40:57,056 | test | WARNING | Thread-57487 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:EaIrOcma7IYrSSY-21-307000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:40:57,063 | test | WARNING | Thread-57461 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:S-21-307000 update failed keys from task: [u'test_collections-3'] 2021-05-18 02:41:08,157 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 02:42:13,917 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 02:42:13,917 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 02:42:41,243 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 02:42:41,243 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 02:43:01,421 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 02:43:01,423 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 02:44:02,246 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1429998875 seconds to complete 2021-05-18 02:44:02,263 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 2881.810s OK During the test, Remote Connections: 162, Disconnections: 162 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 24 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_24 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_25 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in,nodes_init=3,nodes_in=2,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '0', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'update_replica': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_25', 'nodes_init': '3', 'updated_num_replicas': '2', 'GROUP': 'replica_update_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 25, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'replicas'} test_data_load_collections_with_rebalance_in (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 02:44:11,832 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #25 test_data_load_collections_with_rebalance_in ========= 2021-05-18 02:44:12,545 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #25 test_data_load_collections_with_rebalance_in ========= 2021-05-18 02:44:13,071 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x743bfa type=client open_count=1 channel=[id: 0x072f6a9f, 0.0.0.0/0.0.0.0:51637] timeout=300.0>) 2021-05-18 02:44:34,013 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x743bfb type=client open_count=1 channel=[id: 0xf52aadf3, 0.0.0.0/0.0.0.0:43764] timeout=300.0>) 2021-05-18 02:44:36,030 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x743bfc type=client open_count=1 channel=[id: 0xcd3c3436, 0.0.0.0/0.0.0.0:42766] timeout=300.0>) 2021-05-18 02:44:38,049 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x743bfd type=client open_count=1 channel=[id: 0x51f56055, 0.0.0.0/0.0.0.0:45981] timeout=300.0>) 2021-05-18 02:45:02,438 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x743bfe type=client open_count=1 channel=[id: 0x4709cb43, 0.0.0.0/0.0.0.0:43185] timeout=300.0>) 2021-05-18 02:45:04,457 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x743bff type=client open_count=1 channel=[id: 0x8e9dc16d, 0.0.0.0/0.0.0.0:47618] timeout=300.0>) 2021-05-18 02:45:06,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x743c00 type=client open_count=1 channel=[id: 0x7f09d6da, 0.0.0.0/0.0.0.0:45777] timeout=300.0>) 2021-05-18 02:45:30,540 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x743c01 type=client open_count=1 channel=[id: 0x81af5d65, 0.0.0.0/0.0.0.0:43435] timeout=300.0>) 2021-05-18 02:45:32,559 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x743c02 type=client open_count=1 channel=[id: 0x432cf800, 0.0.0.0/0.0.0.0:44162] timeout=300.0>) 2021-05-18 02:45:34,576 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x743c03 type=client open_count=1 channel=[id: 0x00c93215, 0.0.0.0/0.0.0.0:37566] timeout=300.0>) 2021-05-18 02:45:58,469 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x743c04 type=client open_count=1 channel=[id: 0xa2c74f48, 0.0.0.0/0.0.0.0:56137] timeout=300.0>) 2021-05-18 02:46:00,493 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x743c05 type=client open_count=1 channel=[id: 0x6a7a7523, 0.0.0.0/0.0.0.0:35191] timeout=300.0>) 2021-05-18 02:46:02,512 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x743c06 type=client open_count=1 channel=[id: 0xbd349929, 0.0.0.0/0.0.0.0:59209] timeout=300.0>) 2021-05-18 02:46:26,615 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x743c07 type=client open_count=1 channel=[id: 0xbe322117, 0.0.0.0/0.0.0.0:38148] timeout=300.0>) 2021-05-18 02:46:28,632 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x743c08 type=client open_count=1 channel=[id: 0xa57c8d4b, 0.0.0.0/0.0.0.0:50418] timeout=300.0>) 2021-05-18 02:46:30,651 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 02:46:48,371 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 02:47:02,079 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 02:47:02,101 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #25 test_data_load_collections_with_rebalance_in ========= 2021-05-18 02:47:02,138 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #25 test_data_load_collections_with_rebalance_in ========= 2021-05-18 02:47:09,032 | test | INFO | pool-27-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.183333333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-18 02:47:09,059 | test | INFO | pool-27-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 02:47:19,161 | test | INFO | pool-27-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 02:47:19,196 | test | INFO | pool-27-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1629998684 sec 2021-05-18 02:47:19,447 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 02:47:45,499 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 02:47:45,740 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 0 | 629145600 | 176549912 | 34373621 | | bucket2 | couchbase | 0 | none | 0 | 0 | 629145600 | 161791792 | 40489028 | | default | couchbase | 0 | none | 0 | 0 | 6291456000 | 195942360 | 31486819 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-18 02:47:45,740 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 02:48:16,069 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 02:48:16,088 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 02:48:32,075 | test | INFO | pool-27-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................*X | 229.739 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:48:32,213 | test | INFO | pool-27-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****....................................................................................................*X | 29398.723 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 02:48:33,720 | test | INFO | pool-27-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................**X | 223.343 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 02:48:34,006 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 45.378416549 | 6437847040 | 4428169216 | 2277617664 / 7511588864 | 101777 / 0 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 96.090280792 | 6437847040 | 4333252608 | 2368897024 / 7511588864 | 97599 / 0 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.1653590625 | 6437847040 | 3978964992 | 2862727168 / 9530753024 | 99303 / 0 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 02:48:41,553 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 02:49:34,095 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 3000 | 629145600 | 59697080 | 67839747 | | bucket2 | couchbase | 0 | none | 0 | 3000 | 629145600 | 59699536 | 73512383 | | default | couchbase | 0 | none | 0 | 500000 | 6291456000 | 150572776 | 146140310 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 02:49:34,096 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #25 test_data_load_collections_with_rebalance_in ========= 2021-05-18 02:49:40,335 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in 2021-05-18 02:49:40,336 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in 2021-05-18 02:49:40,336 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:459] Updating all the bucket replicas to 2 2021-05-18 02:49:40,640 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 59701464 | 67839747 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 59699536 | 73512383 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 150573256 | 146140310 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 02:49:40,696 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 02:49:46,042 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 02:49:51,272 | test | INFO | pool-27-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 14.5304843495 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 11.2251870865 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 11.4835247254 | Cluster node | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 02:49:51,940 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 02:49:58,022 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 1.79 2021-05-18 02:50:03,674 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.77 2021-05-18 02:50:07,821 | test | INFO | pool-27-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 1774.888 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 02:50:08,901 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 4.62 2021-05-18 02:50:09,278 | test | INFO | pool-27-thread-17 | [table_view:display:72] Ops trend for bucket 'default' +----------+-------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...............X | 4368.818 | | 4368.818 | ..............*.....................................................................................X | 27728.397 | +----------+-------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 02:50:11,049 | test | INFO | pool-27-thread-6 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***..................................................................................**..................*X | 1698.036 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 02:50:14,016 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 7.43 2021-05-18 02:50:19,112 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 12.01 2021-05-18 02:50:24,187 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 17.07 2021-05-18 02:50:29,305 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 21.61 2021-05-18 02:50:34,423 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 25.43 2021-05-18 02:50:39,563 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 29.06 2021-05-18 02:50:44,703 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.74 2021-05-18 02:50:49,779 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 34.92 2021-05-18 02:50:54,921 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.35 2021-05-18 02:51:00,066 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 44.34 2021-05-18 02:51:05,193 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 49.55 2021-05-18 02:51:10,302 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 54.32 2021-05-18 02:51:15,401 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 58.59 2021-05-18 02:51:20,490 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 62.41 2021-05-18 02:51:25,588 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 66.0 2021-05-18 02:51:30,680 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 67.84 2021-05-18 02:51:35,772 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 72.0 2021-05-18 02:51:40,878 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 76.52 2021-05-18 02:51:45,999 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 80.78 2021-05-18 02:51:51,137 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 84.91 2021-05-18 02:51:56,273 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 88.79 2021-05-18 02:52:01,378 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 92.31 2021-05-18 02:52:06,490 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 95.28 2021-05-18 02:52:11,588 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: running, progress: 98.38 2021-05-18 02:52:21,713 | test | INFO | pool-27-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 02:52:21,752 | test | INFO | pool-27-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 150.481000185 sec 2021-05-18 02:52:34,177 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 02:53:54,540 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 02:53:54,540 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 02:54:15,857 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 02:54:15,858 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 02:54:39,951 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 02:54:39,951 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 02:55:38,200 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1349999905 seconds to complete 2021-05-18 02:55:38,217 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 695.969s OK During the test, Remote Connections: 161, Disconnections: 161 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 25 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_25 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_26 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_out,nodes_init=5,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '0', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'update_replica': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_26', 'nodes_init': '5', 'updated_num_replicas': '3', 'GROUP': 'replica_update_with_collection_crud', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 26, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'replicas'} test_data_load_collections_with_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 02:55:47,762 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #26 test_data_load_collections_with_rebalance_out ========= 2021-05-18 02:55:48,427 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #26 test_data_load_collections_with_rebalance_out ========= 2021-05-18 02:55:48,904 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7a0c9f type=client open_count=1 channel=[id: 0xc855573e, 0.0.0.0/0.0.0.0:33905] timeout=300.0>) 2021-05-18 02:56:09,673 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7a0ca0 type=client open_count=1 channel=[id: 0xf067d1dc, 0.0.0.0/0.0.0.0:51429] timeout=300.0>) 2021-05-18 02:56:11,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7a0ca1 type=client open_count=1 channel=[id: 0xcada559b, 0.0.0.0/0.0.0.0:33346] timeout=300.0>) 2021-05-18 02:56:13,710 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7a0ca2 type=client open_count=1 channel=[id: 0xc820a015, 0.0.0.0/0.0.0.0:53281] timeout=300.0>) 2021-05-18 02:56:38,206 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7a0ca3 type=client open_count=1 channel=[id: 0xf9b26c04, 0.0.0.0/0.0.0.0:35512] timeout=300.0>) 2021-05-18 02:56:40,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7a0ca4 type=client open_count=1 channel=[id: 0x569e1727, 0.0.0.0/0.0.0.0:43595] timeout=300.0>) 2021-05-18 02:56:42,244 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7a0ca5 type=client open_count=1 channel=[id: 0xfbac5437, 0.0.0.0/0.0.0.0:47506] timeout=300.0>) 2021-05-18 02:57:06,374 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7a0ca6 type=client open_count=1 channel=[id: 0xc0096eef, 0.0.0.0/0.0.0.0:38808] timeout=300.0>) 2021-05-18 02:57:08,391 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7a0ca7 type=client open_count=1 channel=[id: 0xd3cbfdce, 0.0.0.0/0.0.0.0:43654] timeout=300.0>) 2021-05-18 02:57:10,410 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7a0ca8 type=client open_count=1 channel=[id: 0xf917eae1, 0.0.0.0/0.0.0.0:49005] timeout=300.0>) 2021-05-18 02:57:34,436 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7a0ca9 type=client open_count=1 channel=[id: 0xabdd24a5, 0.0.0.0/0.0.0.0:46461] timeout=300.0>) 2021-05-18 02:57:36,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7a0caa type=client open_count=1 channel=[id: 0xc136979e, 0.0.0.0/0.0.0.0:36434] timeout=300.0>) 2021-05-18 02:57:38,471 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7a0cab type=client open_count=1 channel=[id: 0x04f3248e, 0.0.0.0/0.0.0.0:38246] timeout=300.0>) 2021-05-18 02:58:02,805 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7a0cac type=client open_count=1 channel=[id: 0x8c886268, 0.0.0.0/0.0.0.0:53387] timeout=300.0>) 2021-05-18 02:58:04,823 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7a0cad type=client open_count=1 channel=[id: 0x35540fe4, 0.0.0.0/0.0.0.0:33592] timeout=300.0>) 2021-05-18 02:58:06,841 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 02:58:24,062 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 02:58:37,811 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 02:58:37,832 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #26 test_data_load_collections_with_rebalance_out ========= 2021-05-18 02:58:37,875 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #26 test_data_load_collections_with_rebalance_out ========= 2021-05-18 02:58:51,154 | test | INFO | pool-28-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.38505641761 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 02:58:51,180 | test | INFO | pool-28-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 02:59:01,256 | test | INFO | pool-28-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 02:59:01,286 | test | INFO | pool-28-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1319999695 sec 2021-05-18 02:59:01,539 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 02:59:27,996 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 02:59:28,220 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 0 | 1048576000 | 128938760 | 38564977 | | bucket2 | couchbase | 0 | none | 0 | 0 | 1048576000 | 125520592 | 52208799 | | default | couchbase | 0 | none | 0 | 0 | 10485760000 | 247324184 | 30907427 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-18 02:59:28,220 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 03:00:17,259 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:00:17,276 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:00:33,311 | test | INFO | pool-28-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ****....................................................................................................*X | 55.103 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 03:00:34,704 | test | INFO | pool-28-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ****....................................................................................................**X | 205.510 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 03:00:36,019 | test | INFO | pool-28-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...........................................................*.........................................***X | 24926.518 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:00:36,295 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 92.0443353865 | 6437847040 | 4482289664 | 2243911680 / 7511588864 | 50660 / 0 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 94.8247745199 | 6437847040 | 4438712320 | 2287669248 / 7511588864 | 100200 / 0 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 93.9145338969 | 6437847040 | 4382236672 | 2320703488 / 7511588864 | 100600 / 0 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 96.9761193532 | 6437847040 | 4062650368 | 2769092608 / 9530753024 | 101800 / 0 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 95.3117448043 | 6437847040 | 4477906944 | 2260393984 / 7511588864 | 102200 / 0 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 03:00:48,690 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:02:12,927 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 3000 | 1048576000 | 66353480 | 77713929 | | bucket2 | couchbase | 0 | none | 0 | 3000 | 1048576000 | 66361664 | 91631819 | | default | couchbase | 0 | none | 0 | 500000 | 10485760000 | 157230856 | 126243806 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-18 03:02:12,930 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #26 test_data_load_collections_with_rebalance_out ========= 2021-05-18 03:02:23,063 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_out 2021-05-18 03:02:23,063 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_out 2021-05-18 03:02:23,065 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:410] Updating all the bucket replicas to 3 2021-05-18 03:02:23,388 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 66353480 | 77713929 | | bucket2 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 66362304 | 91631819 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 157231656 | 126243806 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-18 03:02:23,446 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:02:23,585 | test | INFO | pool-28-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 5.80676344606 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 14.74 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 13.466442226 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 14.4535742262 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 11.7448042533 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 03:02:23,621 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:02:28,963 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:02:29,094 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.04 2021-05-18 03:02:34,236 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.77 2021-05-18 03:02:39,463 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.14 2021-05-18 03:02:44,691 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.86 2021-05-18 03:02:45,674 | test | INFO | pool-28-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..............................................................................*......................*X | 587.977 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 03:02:45,719 | test | INFO | pool-28-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **............................................................*........................................*X | 3379.583 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:02:47,539 | test | INFO | pool-28-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........................................................*...............**............................X | 13523.419 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:02:49,788 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 3.76 2021-05-18 03:02:54,871 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 6.35 2021-05-18 03:02:59,950 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 9.0 2021-05-18 03:03:05,023 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 11.79 2021-05-18 03:03:10,095 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 14.87 2021-05-18 03:03:15,214 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 17.92 2021-05-18 03:03:20,298 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 21.22 2021-05-18 03:03:25,365 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 24.28 2021-05-18 03:03:30,457 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 27.55 2021-05-18 03:03:35,539 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.75 2021-05-18 03:03:40,628 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:03:45,690 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:03:50,802 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:03:55,940 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 35.78 2021-05-18 03:04:01,046 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 39.33 2021-05-18 03:04:06,154 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.73 2021-05-18 03:04:11,240 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 46.35 2021-05-18 03:04:16,364 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 49.79 2021-05-18 03:04:21,466 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 53.34 2021-05-18 03:04:26,601 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 57.14 2021-05-18 03:04:31,710 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 60.9 2021-05-18 03:04:36,782 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 64.4 2021-05-18 03:04:41,858 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:04:46,921 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:04:53,036 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 67.1 2021-05-18 03:04:58,118 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.22 2021-05-18 03:05:03,220 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 72.9 2021-05-18 03:05:08,319 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 75.57 2021-05-18 03:05:13,398 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 78.28 2021-05-18 03:05:18,483 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 80.93 2021-05-18 03:05:23,596 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 83.71 2021-05-18 03:05:28,759 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 86.38 2021-05-18 03:05:33,868 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 89.31 2021-05-18 03:05:38,963 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 92.13 2021-05-18 03:05:44,049 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 94.96 2021-05-18 03:05:49,209 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 97.66 2021-05-18 03:05:54,280 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 99.78 2021-05-18 03:05:59,395 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:06:04,460 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:06:14,562 | test | INFO | pool-28-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:06:14,732 | test | INFO | pool-28-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 231.147000074 sec 2021-05-18 03:06:23,438 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:07:13,920 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 03:07:13,921 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 03:07:37,752 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 03:07:37,752 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 03:07:52,943 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 03:07:52,943 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 03:09:02,647 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1270000935 seconds to complete 2021-05-18 03:09:02,661 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 804.350s OK During the test, Remote Connections: 163, Disconnections: 163 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 26 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_26 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_27 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_swap_rebalance,nodes_init=3,nodes_swap=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '1', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'update_replica': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_27', 'nodes_init': '3', 'updated_num_replicas': '3', 'GROUP': 'replica_update_with_collection_crud', 'nodes_swap': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 27, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'replicas'} test_data_load_collections_with_swap_rebalance (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 03:09:12,114 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #27 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 03:09:12,806 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #27 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 03:09:13,329 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7fdb37 type=client open_count=1 channel=[id: 0x03ff324f, 0.0.0.0/0.0.0.0:60489] timeout=300.0>) 2021-05-18 03:09:34,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7fdb38 type=client open_count=1 channel=[id: 0x70e3ef24, 0.0.0.0/0.0.0.0:48775] timeout=300.0>) 2021-05-18 03:09:36,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7fdb39 type=client open_count=1 channel=[id: 0xa1101030, 0.0.0.0/0.0.0.0:37761] timeout=300.0>) 2021-05-18 03:09:38,140 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7fdb3a type=client open_count=1 channel=[id: 0x4314c574, 0.0.0.0/0.0.0.0:52781] timeout=300.0>) 2021-05-18 03:10:02,323 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7fdb3b type=client open_count=1 channel=[id: 0x075ef307, 0.0.0.0/0.0.0.0:40414] timeout=300.0>) 2021-05-18 03:10:04,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7fdb3c type=client open_count=1 channel=[id: 0xa8e2f0f3, 0.0.0.0/0.0.0.0:38475] timeout=300.0>) 2021-05-18 03:10:06,361 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7fdb3d type=client open_count=1 channel=[id: 0x9bd152bc, 0.0.0.0/0.0.0.0:46275] timeout=300.0>) 2021-05-18 03:10:30,403 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7fdb3e type=client open_count=1 channel=[id: 0x3e277730, 0.0.0.0/0.0.0.0:41709] timeout=300.0>) 2021-05-18 03:10:32,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7fdb3f type=client open_count=1 channel=[id: 0x0c39b5a6, 0.0.0.0/0.0.0.0:57467] timeout=300.0>) 2021-05-18 03:10:34,444 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7fdb40 type=client open_count=1 channel=[id: 0x81b23b0d, 0.0.0.0/0.0.0.0:47618] timeout=300.0>) 2021-05-18 03:10:58,665 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7fdb41 type=client open_count=1 channel=[id: 0xb3532b8e, 0.0.0.0/0.0.0.0:39678] timeout=300.0>) 2021-05-18 03:11:00,684 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7fdb42 type=client open_count=1 channel=[id: 0x17b8c87d, 0.0.0.0/0.0.0.0:55191] timeout=300.0>) 2021-05-18 03:11:02,700 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7fdb43 type=client open_count=1 channel=[id: 0x0a12e28d, 0.0.0.0/0.0.0.0:54523] timeout=300.0>) 2021-05-18 03:11:27,118 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7fdb44 type=client open_count=1 channel=[id: 0x4d27954d, 0.0.0.0/0.0.0.0:50143] timeout=300.0>) 2021-05-18 03:11:29,138 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7fdb45 type=client open_count=1 channel=[id: 0x89f78aed, 0.0.0.0/0.0.0.0:38107] timeout=300.0>) 2021-05-18 03:11:31,157 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 03:11:48,404 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 03:12:02,158 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 03:12:02,186 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #27 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 03:12:02,223 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #27 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 03:12:08,891 | test | INFO | pool-29-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.291642363136 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-18 03:12:08,917 | test | INFO | pool-29-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:12:18,989 | test | INFO | pool-29-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:12:19,015 | test | INFO | pool-29-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.125 sec 2021-05-18 03:12:19,273 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 03:12:57,947 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 03:12:58,239 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 0 | 629145600 | 535500152 | 49506499 | | bucket2 | couchbase | 1 | none | 0 | 0 | 629145600 | 507977152 | 59405597 | | default | couchbase | 1 | none | 0 | 0 | 6291456000 | 555032536 | 34863724 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-18 03:12:58,240 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 03:13:26,798 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:13:26,806 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:13:45,479 | test | INFO | pool-29-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................**X | 174.924 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 03:13:45,671 | test | INFO | pool-29-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | ***....................................................................................................**X | 31.487 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 03:13:45,805 | test | INFO | pool-29-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................**X | 22990.550 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:13:46,078 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 99.8194402648 | 6437847040 | 4103249920 | 2619502592 / 7511588864 | 79386 / 61317 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9216666667 | 6437847040 | 4096757760 | 2623660032 / 7511588864 | 75798 / 67431 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.5576403031 | 6437847040 | 3780055040 | 3061018624 / 9530753024 | 76805 / 53240 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 03:13:57,951 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:14:50,217 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 3000 | 629145600 | 108402248 | 134473822 | | bucket2 | couchbase | 1 | none | 0 | 3000 | 629145600 | 107678928 | 142440881 | | default | couchbase | 1 | none | 0 | 500000 | 6291456000 | 360448024 | 307771838 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:14:50,219 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #27 test_data_load_collections_with_swap_rebalance ========= 2021-05-18 03:14:56,519 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during swap_rebalance 2021-05-18 03:14:56,519 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : swap_rebalance 2021-05-18 03:14:56,519 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:512] Updating all the bucket replicas to 3 2021-05-18 03:14:56,867 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 629145600 | 108402248 | 134473822 | | bucket2 | couchbase | 3 | none | 0 | 3000 | 629145600 | 107678928 | 142440881 | | default | couchbase | 3 | none | 0 | 500000 | 6291456000 | 360448184 | 307771838 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:15:04,321 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:15:04,457 | test | INFO | pool-29-thread-26 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 18.88 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 14.1668469488 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 14.7133333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.390006500108 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-18 03:15:04,510 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:15:09,746 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:15:14,155 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:15:15,148 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.83 2021-05-18 03:15:20,576 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 1.69 2021-05-18 03:15:26,010 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.23 2021-05-18 03:15:31,196 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.99 2021-05-18 03:15:34,887 | test | INFO | pool-29-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **................................................................................................*....X | 3076.927 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:15:36,361 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 4.52 2021-05-18 03:15:37,338 | test | INFO | pool-29-thread-29 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................**X | 3062.730 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:15:39,513 | test | INFO | pool-29-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.........................................................................................***...........X | 20186.575 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:15:41,539 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 6.31 2021-05-18 03:15:46,661 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 8.11 2021-05-18 03:15:51,742 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 10.05 2021-05-18 03:15:56,845 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 12.34 2021-05-18 03:16:01,967 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.98 2021-05-18 03:16:07,072 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 17.75 2021-05-18 03:16:12,163 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 20.8 2021-05-18 03:16:17,253 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 24.03 2021-05-18 03:16:22,371 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 27.62 2021-05-18 03:16:27,440 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.12 2021-05-18 03:16:32,515 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:16:37,578 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:16:44,325 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:16:49,441 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 37.64 2021-05-18 03:16:54,543 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 41.11 2021-05-18 03:16:59,701 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-18 03:17:04,811 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 47.69 2021-05-18 03:17:09,946 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 51.01 2021-05-18 03:17:15,118 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 54.39 2021-05-18 03:17:20,221 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 57.5 2021-05-18 03:17:25,309 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 61.02 2021-05-18 03:17:30,463 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 64.29 2021-05-18 03:17:35,589 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:17:40,658 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:17:45,756 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:17:50,877 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 68.62 2021-05-18 03:17:55,964 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 71.44 2021-05-18 03:18:01,102 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 74.04 2021-05-18 03:18:06,282 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 76.65 2021-05-18 03:18:11,421 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 79.25 2021-05-18 03:18:16,519 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 81.86 2021-05-18 03:18:21,618 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 84.53 2021-05-18 03:18:26,717 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 87.02 2021-05-18 03:18:31,855 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 89.5 2021-05-18 03:18:36,992 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 92.14 2021-05-18 03:18:42,065 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 94.6 2021-05-18 03:18:47,167 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 97.1 2021-05-18 03:18:52,263 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 99.27 2021-05-18 03:18:57,338 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:19:02,400 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:19:12,483 | test | INFO | pool-29-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:19:12,634 | test | INFO | pool-29-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 248.177000046 sec 2021-05-18 03:19:21,165 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:20:10,670 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 03:20:10,670 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 03:20:30,940 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 03:20:30,941 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 03:20:52,441 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 03:20:52,441 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 03:22:03,721 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1429998875 seconds to complete 2021-05-18 03:22:03,740 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 781.061s OK During the test, Remote Connections: 131, Disconnections: 131 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 27 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_27 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_28 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=3,nodes_in=2,nodes_out=1,update_replica=True,updated_num_replicas=2,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=1,skip_validations=False,GROUP=replica_update_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '1', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '2', 'update_replica': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_28', 'nodes_init': '3', 'updated_num_replicas': '2', 'GROUP': 'replica_update_with_collection_crud', 'nodes_out': '1', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 28, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 03:22:13,944 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #28 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:22:14,713 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #28 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:22:15,319 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x85ac12 type=client open_count=1 channel=[id: 0x135bbec8, 0.0.0.0/0.0.0.0:32854] timeout=300.0>) 2021-05-18 03:22:36,496 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x85ac13 type=client open_count=1 channel=[id: 0x1323bc91, 0.0.0.0/0.0.0.0:42644] timeout=300.0>) 2021-05-18 03:22:38,513 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x85ac14 type=client open_count=1 channel=[id: 0x1d546c95, 0.0.0.0/0.0.0.0:42692] timeout=300.0>) 2021-05-18 03:22:40,536 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x85ac15 type=client open_count=1 channel=[id: 0xb780efc2, 0.0.0.0/0.0.0.0:41023] timeout=300.0>) 2021-05-18 03:23:05,029 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x85ac16 type=client open_count=1 channel=[id: 0xe6850898, 0.0.0.0/0.0.0.0:49456] timeout=300.0>) 2021-05-18 03:23:07,051 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x85ac17 type=client open_count=1 channel=[id: 0xffd402cd, 0.0.0.0/0.0.0.0:54749] timeout=300.0>) 2021-05-18 03:23:09,069 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x85ac18 type=client open_count=1 channel=[id: 0x5891363e, 0.0.0.0/0.0.0.0:39263] timeout=300.0>) 2021-05-18 03:23:33,653 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x85ac19 type=client open_count=1 channel=[id: 0xf7a0b159, 0.0.0.0/0.0.0.0:45785] timeout=300.0>) 2021-05-18 03:23:35,673 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x85ac1a type=client open_count=1 channel=[id: 0xf9665d03, 0.0.0.0/0.0.0.0:51846] timeout=300.0>) 2021-05-18 03:23:37,690 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x85ac1b type=client open_count=1 channel=[id: 0x557a04a7, 0.0.0.0/0.0.0.0:42218] timeout=300.0>) 2021-05-18 03:24:02,005 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x85ac1c type=client open_count=1 channel=[id: 0x0932e788, 0.0.0.0/0.0.0.0:56718] timeout=300.0>) 2021-05-18 03:24:04,022 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x85ac1d type=client open_count=1 channel=[id: 0xcb028775, 0.0.0.0/0.0.0.0:58613] timeout=300.0>) 2021-05-18 03:24:06,039 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x85ac1e type=client open_count=1 channel=[id: 0x5cce42ac, 0.0.0.0/0.0.0.0:45759] timeout=300.0>) 2021-05-18 03:24:30,023 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x85ac1f type=client open_count=1 channel=[id: 0xf7697319, 0.0.0.0/0.0.0.0:60973] timeout=300.0>) 2021-05-18 03:24:32,040 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x85ac20 type=client open_count=1 channel=[id: 0xcd1f179f, 0.0.0.0/0.0.0.0:35443] timeout=300.0>) 2021-05-18 03:24:34,059 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 03:24:51,536 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 03:25:05,313 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 03:25:05,335 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #28 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:25:05,374 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #28 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:25:12,612 | test | INFO | pool-30-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 9.998499975 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | +----------------+----------+-----------------------+-------------+--------------+ 2021-05-18 03:25:12,635 | test | INFO | pool-30-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:25:22,707 | test | INFO | pool-30-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:25:22,736 | test | INFO | pool-30-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-18 03:25:22,993 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 03:25:59,188 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 03:25:59,747 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 0 | 629145600 | 553145624 | 40023244 | | bucket2 | couchbase | 1 | none | 0 | 0 | 629145600 | 532016880 | 50029758 | | default | couchbase | 1 | none | 0 | 0 | 6291456000 | 654447688 | 29341848 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-18 03:25:59,747 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 03:26:29,089 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:26:29,098 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:26:49,881 | test | INFO | pool-30-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **.............*...........................**............................................................X | 67.019 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 03:26:51,017 | test | INFO | pool-30-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....................*............................................***...................................X | 142.625 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 03:26:51,778 | test | INFO | pool-30-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................*.....................................***.............................................*X | 25300.619 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:26:52,036 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 95.143388828 | 6437847040 | 4149575680 | 2571976704 / 7511588864 | 129632 / 129088 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 96.5563289766 | 6437847040 | 4136529920 | 2576404480 / 7511588864 | 135766 / 133010 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 95.1775039241 | 6437847040 | 3745685504 | 3095678976 / 9530753024 | 131476 / 130436 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 03:27:02,720 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:27:57,450 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 3000 | 629145600 | 107905656 | 129283259 | | bucket2 | couchbase | 1 | none | 0 | 3000 | 629145600 | 107713904 | 144124236 | | default | couchbase | 1 | none | 0 | 500000 | 6291456000 | 364219080 | 306948529 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:27:57,450 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #28 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:28:04,140 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-18 03:28:04,140 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-18 03:28:04,141 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:571] Updating all the bucket replicas to 2 2021-05-18 03:28:04,519 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 629145600 | 107905656 | 129283259 | | bucket2 | couchbase | 2 | none | 0 | 3000 | 629145600 | 107713904 | 144124236 | | default | couchbase | 2 | none | 0 | 500000 | 6291456000 | 364219080 | 306948529 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:28:11,565 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:28:11,740 | test | INFO | pool-30-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 11.7418623644 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 9.115 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 18.045300755 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 03:28:11,799 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:28:16,914 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:28:20,835 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:28:22,099 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.39 2021-05-18 03:28:27,309 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.09 2021-05-18 03:28:32,855 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.43 2021-05-18 03:28:38,079 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 3.02 2021-05-18 03:28:38,674 | test | INFO | pool-30-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................*X | 1125.950 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:28:38,786 | test | INFO | pool-30-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................................................................................................*..X | 1177.868 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:28:38,868 | test | INFO | pool-30-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........................*.........................................................................X | 20924.505 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:28:43,246 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 5.56 2021-05-18 03:28:48,359 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 7.84 2021-05-18 03:28:53,460 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 10.49 2021-05-18 03:28:58,555 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 13.42 2021-05-18 03:29:03,664 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.79 2021-05-18 03:29:08,740 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 20.93 2021-05-18 03:29:13,851 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 24.15 2021-05-18 03:29:18,940 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 27.34 2021-05-18 03:29:24,029 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 30.71 2021-05-18 03:29:29,135 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:29:34,197 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:29:39,270 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:29:44,437 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 36.27 2021-05-18 03:29:49,565 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 41.31 2021-05-18 03:29:54,657 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 46.29 2021-05-18 03:29:59,769 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 51.19 2021-05-18 03:30:04,845 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.52 2021-05-18 03:30:09,993 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 59.55 2021-05-18 03:30:15,091 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 63.24 2021-05-18 03:30:20,177 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.18 2021-05-18 03:30:25,243 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:30:30,302 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:30:35,413 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 67.64 2021-05-18 03:30:40,512 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 71.81 2021-05-18 03:30:45,582 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 75.32 2021-05-18 03:30:50,680 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 79.14 2021-05-18 03:30:55,788 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 83.07 2021-05-18 03:31:00,888 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 86.68 2021-05-18 03:31:05,999 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 90.08 2021-05-18 03:31:11,101 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 93.5 2021-05-18 03:31:16,187 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 96.01 2021-05-18 03:31:21,259 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 98.24 2021-05-18 03:31:26,334 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:31:31,394 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:31:37,181 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:31:47,282 | test | INFO | pool-30-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:31:47,382 | test | INFO | pool-30-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 215.643999815 sec 2021-05-18 03:31:57,874 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:33:02,960 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 03:33:02,960 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 03:33:21,510 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 03:33:21,512 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 03:33:42,585 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 03:33:42,585 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 03:34:44,345 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.2020001411 seconds to complete 2021-05-18 03:34:44,359 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 760.590s OK During the test, Remote Connections: 146, Disconnections: 146 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 28 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_28 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_29 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_rebalance_in_out,nodes_init=4,nodes_in=1,nodes_out=2,update_replica=True,updated_num_replicas=3,bucket_spec=multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections,data_load_stage=during,data_load_spec=volume_test_load_with_CRUD_on_collections,skip_validations=False,override_spec_params=replicas,replicas=0,skip_validations=False,GROUP=replica_update_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '0', 'conf_file': 'conf/collections/collections_rebalance_crud_on_collections.conf', 'crash_warning': 'True', 'nodes_in': '1', 'update_replica': 'True', 'skip_validations': 'False', 'spec': 'collections_rebalance_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_29', 'nodes_init': '4', 'updated_num_replicas': '3', 'GROUP': 'replica_update_with_collection_crud', 'nodes_out': '2', 'bucket_spec': 'multi_bucket.buckets_all_membase_for_rebalance_tests_more_collections', 'case_number': 29, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'replicas'} test_data_load_collections_with_rebalance_in_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-18 03:34:53,914 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #29 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:34:54,605 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #29 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:34:55,039 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8b6f37 type=client open_count=1 channel=[id: 0xafdda5a6, 0.0.0.0/0.0.0.0:37290] timeout=300.0>) 2021-05-18 03:35:15,815 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8b6f38 type=client open_count=1 channel=[id: 0x7c095fc6, 0.0.0.0/0.0.0.0:58843] timeout=300.0>) 2021-05-18 03:35:17,839 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x8b6f39 type=client open_count=1 channel=[id: 0xd337e104, 0.0.0.0/0.0.0.0:59854] timeout=300.0>) 2021-05-18 03:35:19,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8b6f3a type=client open_count=1 channel=[id: 0x1cd275b7, 0.0.0.0/0.0.0.0:42691] timeout=300.0>) 2021-05-18 03:35:43,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8b6f3b type=client open_count=1 channel=[id: 0xca9816fe, 0.0.0.0/0.0.0.0:43501] timeout=300.0>) 2021-05-18 03:35:45,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x8b6f3c type=client open_count=1 channel=[id: 0xce472415, 0.0.0.0/0.0.0.0:48701] timeout=300.0>) 2021-05-18 03:35:47,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8b6f3d type=client open_count=1 channel=[id: 0x9f17029a, 0.0.0.0/0.0.0.0:46323] timeout=300.0>) 2021-05-18 03:36:11,990 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8b6f3e type=client open_count=1 channel=[id: 0x96e6fe44, 0.0.0.0/0.0.0.0:45480] timeout=300.0>) 2021-05-18 03:36:14,007 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x8b6f3f type=client open_count=1 channel=[id: 0x4a6229b4, 0.0.0.0/0.0.0.0:58835] timeout=300.0>) 2021-05-18 03:36:16,026 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8b6f40 type=client open_count=1 channel=[id: 0xbd2ab6f5, 0.0.0.0/0.0.0.0:33452] timeout=300.0>) 2021-05-18 03:36:40,232 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8b6f41 type=client open_count=1 channel=[id: 0x07434545, 0.0.0.0/0.0.0.0:52440] timeout=300.0>) 2021-05-18 03:36:42,249 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x8b6f42 type=client open_count=1 channel=[id: 0x85ee41d7, 0.0.0.0/0.0.0.0:59081] timeout=300.0>) 2021-05-18 03:36:44,267 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8b6f43 type=client open_count=1 channel=[id: 0xcf00f0ad, 0.0.0.0/0.0.0.0:50462] timeout=300.0>) 2021-05-18 03:37:08,209 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8b6f44 type=client open_count=1 channel=[id: 0xea0960ee, 0.0.0.0/0.0.0.0:52421] timeout=300.0>) 2021-05-18 03:37:10,226 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x8b6f45 type=client open_count=1 channel=[id: 0x3e48a3b1, 0.0.0.0/0.0.0.0:56444] timeout=300.0>) 2021-05-18 03:37:12,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-18 03:37:29,296 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-18 03:37:43,026 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-18 03:37:43,046 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #29 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:37:43,082 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #29 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:37:53,457 | test | INFO | pool-31-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.338327694538 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-18 03:37:53,480 | test | INFO | pool-31-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:38:03,552 | test | INFO | pool-31-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:38:03,578 | test | INFO | pool-31-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1199998856 sec 2021-05-18 03:38:03,864 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-18 03:38:32,328 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-18 03:38:32,555 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 0 | 838860800 | 118369424 | 50988676 | | bucket2 | couchbase | 0 | none | 0 | 0 | 838860800 | 119425616 | 58164721 | | default | couchbase | 0 | none | 0 | 0 | 8388608000 | 121069168 | 41275958 | +---------+-----------+----------+------------+-----+-------+------------+-----------+-----------+ 2021-05-18 03:38:32,556 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-18 03:39:09,917 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:39:09,927 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:39:24,463 | test | INFO | pool-31-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................**X | 16250.302 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-18 03:39:25,654 | test | INFO | pool-31-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................***X | 104.078 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-18 03:39:26,009 | test | INFO | pool-31-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **....................................................................................................***X | 33.593 | +-------+------------------------------------------------------------------------------------------------------------+--------+ 2021-05-18 03:39:26,262 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.127 | kv | 96.9816163603 | 6437847040 | 4390699008 | 2314399744 / 7511588864 | 42923 / 0 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 97.815198762 | 6437847040 | 4391301120 | 2304348160 / 7511588864 | 39641 / 0 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 97.9918005466 | 6437847040 | 4013637632 | 2799316992 / 9530753024 | 41475 / 0 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 95.96340061 | 6437847040 | 4435210240 | 2277158912 / 7511588864 | 39844 / 0 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-18 03:39:35,749 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:40:43,516 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 0 | none | 0 | 3000 | 838860800 | 63021264 | 82989807 | | bucket2 | couchbase | 0 | none | 0 | 3000 | 838860800 | 63032848 | 90677832 | | default | couchbase | 0 | none | 0 | 500000 | 8388608000 | 153897936 | 138204460 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:40:43,516 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #29 test_data_load_collections_with_rebalance_in_out ========= 2021-05-18 03:40:51,638 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during rebalance_in_out 2021-05-18 03:40:51,640 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : rebalance_in_out 2021-05-18 03:40:51,640 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:571] Updating all the bucket replicas to 3 2021-05-18 03:40:51,976 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 838860800 | 63021264 | 82989807 | | bucket2 | couchbase | 3 | none | 0 | 3000 | 838860800 | 63028464 | 90677832 | | default | couchbase | 3 | none | 0 | 500000 | 8388608000 | 153897616 | 138204460 | +---------+-----------+----------+------------+-----+--------+------------+-----------+-----------+ 2021-05-18 03:40:55,854 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-18 03:40:55,980 | test | INFO | pool-31-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | unknown | None | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 14.2142809521 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 13.465993367 | --- OUT ---> | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 15.1580806987 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 13.7014383094 | --- OUT ---> | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-18 03:40:56,010 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:41:01,101 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-18 03:41:02,092 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-18 03:41:06,279 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.25 2021-05-18 03:41:11,539 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.61 2021-05-18 03:41:16,667 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 1.46 2021-05-18 03:41:17,255 | test | INFO | pool-31-thread-30 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................*X | 2218.579 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:41:19,040 | test | INFO | pool-31-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......................**..............................................................................X | 2725.645 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:41:19,108 | test | INFO | pool-31-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *........................................**............................................................X | 2218.060 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-18 03:41:21,743 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 3.06 2021-05-18 03:41:26,835 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 6.08 2021-05-18 03:41:31,917 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 8.91 2021-05-18 03:41:37,032 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 12.18 2021-05-18 03:41:42,111 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 15.67 2021-05-18 03:41:47,220 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 19.08 2021-05-18 03:41:52,371 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 22.67 2021-05-18 03:41:57,453 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 26.32 2021-05-18 03:42:02,530 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 30.01 2021-05-18 03:42:07,602 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:42:12,664 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:42:17,753 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-18 03:42:22,980 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 36.17 2021-05-18 03:42:28,065 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 39.8 2021-05-18 03:42:33,170 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 42.99 2021-05-18 03:42:38,266 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 46.46 2021-05-18 03:42:43,359 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 49.73 2021-05-18 03:42:48,441 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 53.36 2021-05-18 03:42:53,536 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 57.12 2021-05-18 03:42:58,645 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 60.76 2021-05-18 03:43:03,750 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 64.85 2021-05-18 03:43:08,848 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:43:13,921 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-18 03:43:19,559 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 67.12 2021-05-18 03:43:24,694 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.24 2021-05-18 03:43:29,796 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 73.06 2021-05-18 03:43:34,877 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 75.6 2021-05-18 03:43:40,003 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 78.37 2021-05-18 03:43:45,098 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 81.26 2021-05-18 03:43:50,217 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 84.26 2021-05-18 03:43:55,313 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 87.07 2021-05-18 03:44:00,446 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 89.87 2021-05-18 03:44:05,588 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 92.78 2021-05-18 03:44:10,690 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 95.5 2021-05-18 03:44:15,792 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 98.41 2021-05-18 03:44:20,875 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:44:25,944 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-18 03:44:36,029 | test | INFO | pool-31-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-18 03:44:36,180 | test | INFO | pool-31-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 220.199999809 sec 2021-05-18 03:44:44,887 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-18 03:45:34,151 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-18 03:45:34,151 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-18 03:45:56,055 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-18 03:45:56,055 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-18 03:46:15,625 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-18 03:46:15,625 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-18 03:47:24,867 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-18 03:47:24,881 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 760.471s OK During the test, Remote Connections: 147, Disconnections: 147 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 29 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_17-24-58/test_29 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 10h 22m 42s 2 actionable tasks: 1 executed, 1 up-to-date Archiving artifacts Recording test results [description-setter] Description set: 1.1.7 -P client_version=default -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest' Notifying upstream projects of job completion Finished: SUCCESS