Started by user Balakumaran G [EnvInject] - Loading node environment variables. Building remotely on slv-s22501 (P0 jython_slave PY3P0) in workspace /data/workspace/temp_win_private_job_3 [WS-CLEANUP] Deleting project workspace... [WS-CLEANUP] Done Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF > /usr/bin/git init /data/workspace/temp_win_private_job_3 # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision 5211d4c425c2693f9ed4de320e82d1c6b0873f86 (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 5211d4c425c2693f9ed4de320e82d1c6b0873f86 > /usr/bin/git rev-list 5211d4c425c2693f9ed4de320e82d1c6b0873f86 # timeout=10 [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins8373584077483475821.sh + echo '[global] username:Administrator password:Membase123 index_port:9102 n1ql_port:8093 [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 [_1] ip:172.23.107.126 port:8091 [_2] ip:172.23.107.127 port:8091 [_3] ip:172.23.107.129 port:8091 [_4] ip:172.23.107.131 port:8091 [_5] ip:172.23.107.142 port:8091' [temp_win_private_job_3] $ /bin/sh -xe /tmp/jenkins6382211605629233606.sh + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + jython_path=/opt/jython/bin/jython + guides/gradlew check -s Starting a Gradle Daemon, 5 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.0 > Task :compileJava Note: /data/workspace/temp_win_private_job_3/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :processResources > Task :classes > Task :compileTestJava NO-SOURCE > Task :processTestResources NO-SOURCE > Task :testClasses UP-TO-DATE > Task :test NO-SOURCE > Task :check UP-TO-DATE Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 12s 2 actionable tasks: 2 executed + guides/gradlew -v ------------------------------------------------------------ Gradle 6.2.2 ------------------------------------------------------------ Build time: 2020-03-04 08:49:31 UTC Revision: 7d0bf6dcb46c143bcc3b7a0fa40a8e5ca28e5856 Kotlin: 1.3.61 Groovy: 2.5.8 Ant: Apache Ant(TM) version 1.10.7 compiled on September 1 2019 JVM: 1.8.0_252 (Oracle Corporation 25.252-b09) OS: Linux 3.10.0-1127.13.1.el7.x86_64 amd64 + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.7 -P client_version=default -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest' > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.7 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/ssh.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar scripts/eagles_all_around.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main scripts/install.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_win_private_job_3/build/classes/java/main:/data/workspace/temp_win_private_job_3/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.7/8ff63cf832199200625bf463c30c9fc36a073480/couchbase-transactions-1.1.7.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.4/7c33fa5f716408f201bd4006532cb97746b2eba5/java-client-3.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/tracing-opentelemetry/0.3.4/c6eadab1abddeba36c597fbe59090e683c966b3c/tracing-opentelemetry-0.3.4.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.4/faa9a317d72193b2956df822d5ac808e463ac70e/core-io-2.1.4.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.5/c8adeb48f74b16c2dfb4f06a880820b9eaed50cf/reactor-core-3.4.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-api/1.0.1/811e28418b73c96de491d71a22d9cb10074870a1/opentelemetry-api-1.0.1.jar:/root/.gradle/caches/modules-2/files-2.1/io.opentelemetry/opentelemetry-context/1.0.1/96319166ccabed5364d70ce2414e89b0c86682a7/opentelemetry-context-1.0.1.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/testexec-temp_win_private_job_3.ini -p rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -c conf/collections/collections_failover_crud_on_collections.conf -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: conf/collections/collections_failover_crud_on_collections.conf Prefix: bucket_collections.collections_rebalance.CollectionsRebalance Global Test input params: {'cluster_name': 'testexec-temp_win_private_job_3', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'get-cbcollect-info': 'True', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'num_nodes': 5, 'quota_percent': '95', 'rerun': 'False', 'spec': 'collections_failover_crud_on_collections'} Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 1, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:18:23,361 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:18:24,331 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:18:27,602 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xc type=client open_count=1 channel=[id: 0xd22b6a3d, 0.0.0.0/0.0.0.0:37310] timeout=300.0>) 2021-05-17 00:18:49,316 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xc8e444c8, 0.0.0.0/0.0.0.0:59249] timeout=300.0>) 2021-05-17 00:18:51,357 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x0a25d28f, 0.0.0.0/0.0.0.0:52035] timeout=300.0>) 2021-05-17 00:18:53,395 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xc5a796ed, 0.0.0.0/0.0.0.0:33020] timeout=300.0>) 2021-05-17 00:19:17,285 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xeab81305, 0.0.0.0/0.0.0.0:44047] timeout=300.0>) 2021-05-17 00:19:19,319 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0x7bc34bf8, 0.0.0.0/0.0.0.0:60958] timeout=300.0>) 2021-05-17 00:19:21,355 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x12 type=client open_count=1 channel=[id: 0x91416262, 0.0.0.0/0.0.0.0:43846] timeout=300.0>) 2021-05-17 00:19:23,382 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x13 type=client open_count=1 channel=[id: 0xa200be5a, 0.0.0.0/0.0.0.0:46957] timeout=300.0>) 2021-05-17 00:19:25,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x14 type=client open_count=1 channel=[id: 0xcd21a2fd, 0.0.0.0/0.0.0.0:47241] timeout=300.0>) 2021-05-17 00:19:50,178 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x15 type=client open_count=1 channel=[id: 0x40fcbeb6, 0.0.0.0/0.0.0.0:34267] timeout=300.0>) 2021-05-17 00:19:52,206 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x16 type=client open_count=1 channel=[id: 0x8d1c475a, 0.0.0.0/0.0.0.0:47931] timeout=300.0>) 2021-05-17 00:19:54,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x17 type=client open_count=1 channel=[id: 0x6ad961f2, 0.0.0.0/0.0.0.0:51299] timeout=300.0>) 2021-05-17 00:19:56,260 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x18 type=client open_count=1 channel=[id: 0x97148c38, 0.0.0.0/0.0.0.0:56528] timeout=300.0>) 2021-05-17 00:19:58,313 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x19 type=client open_count=1 channel=[id: 0x84c9651f, 0.0.0.0/0.0.0.0:60804] timeout=300.0>) 2021-05-17 00:20:23,196 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1a type=client open_count=1 channel=[id: 0x8a8aa2e7, 0.0.0.0/0.0.0.0:45262] timeout=300.0>) 2021-05-17 00:20:25,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1b type=client open_count=1 channel=[id: 0x6fc702da, 0.0.0.0/0.0.0.0:57773] timeout=300.0>) 2021-05-17 00:20:27,255 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c type=client open_count=1 channel=[id: 0x32dcb4ff, 0.0.0.0/0.0.0.0:48925] timeout=300.0>) 2021-05-17 00:20:29,282 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1d type=client open_count=1 channel=[id: 0x39368e22, 0.0.0.0/0.0.0.0:46092] timeout=300.0>) 2021-05-17 00:20:31,309 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1e type=client open_count=1 channel=[id: 0xfd46a090, 0.0.0.0/0.0.0.0:52959] timeout=300.0>) 2021-05-17 00:20:55,869 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1f type=client open_count=1 channel=[id: 0x620cafeb, 0.0.0.0/0.0.0.0:53061] timeout=300.0>) 2021-05-17 00:20:57,895 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x20 type=client open_count=1 channel=[id: 0xa4aad489, 0.0.0.0/0.0.0.0:58083] timeout=300.0>) 2021-05-17 00:20:59,921 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x21 type=client open_count=1 channel=[id: 0xa1fdc501, 0.0.0.0/0.0.0.0:39363] timeout=300.0>) 2021-05-17 00:21:01,948 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x22 type=client open_count=1 channel=[id: 0x44123ef1, 0.0.0.0/0.0.0.0:60660] timeout=300.0>) 2021-05-17 00:21:03,976 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:21:22,219 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:21:36,270 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:21:36,309 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:21:36,369 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:21:52,003 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.56833333333 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 00:21:52,046 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:22:02,173 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:22:02,216 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2150001526 sec 2021-05-17 00:22:02,625 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 00:22:49,197 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 00:22:49,601 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 993609288 | 107565129 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 301356080 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1288556936 | 51935309 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 00:22:49,602 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 00:23:41,921 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:23:41,963 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:24:07,217 | test | INFO | pool-2-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..............................***......................................................................*X | 121.151 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:24:09,006 | test | INFO | pool-2-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....***.................................................................................**.............X | 25614.782 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:24:09,823 | test | INFO | pool-2-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *........................**........................................................................**....*X | 135.670 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:24:10,473 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9727803202 | 6437847040 | 4270555136 | 2563874816 / 7511588864 | 100098 / 271488 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.663153682 | 6437847040 | 4244692992 | 2576601088 / 7511588864 | 98760 / 255949 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8435835996 | 6437847040 | 4263268352 | 2580705280 / 7511588864 | 97692 / 284744 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.7930055922 | 6437847040 | 3888656384 | 3154681856 / 9530753024 | 99763 / 258387 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8951677316 | 6437847040 | 4261179392 | 2571583488 / 7511588864 | 98812 / 255994 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 00:24:23,805 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:25:51,835 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 213250248 | 296259731 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 318381232 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 848323416 | 628598626 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 00:25:51,836 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #1 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:26:02,401 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 00:26:04,026 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 00:26:04,028 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_rebalance_out 2021-05-17 00:26:04,035 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:26:13,194 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:26:13,463 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 00:26:13,463 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 00:26:30,509 | test | INFO | pool-2-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................................................................................X | 19863.163 | +-------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:26:30,846 | test | INFO | pool-2-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 6139.082 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:27:40,743 | test | INFO | pool-2-thread-29 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 86.7849998474 seconds to complete 2021-05-17 00:27:40,750 | test | INFO | pool-2-thread-29 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:29:45,819 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0629999637604 seconds 2021-05-17 00:30:34,052 | test | INFO | pool-2-thread-17 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 48.0609998703 seconds to complete 2021-05-17 00:30:34,059 | test | INFO | pool-2-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:32:39,121 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 00:32:39,305 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 00:32:39,308 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:32:45,446 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:33:00,361 | test | INFO | pool-2-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5044.908 | | 4932.970 | ................................................................................................X | 4932.970 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:33:00,828 | test | INFO | pool-2-thread-20 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................................*................................................X | 21149.525 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:33:01,865 | test | INFO | pool-2-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 6100.229 | | 2389.509 | ......................................*X | 2389.509 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:33:02,145 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,147 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,197 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:33:02,197 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:33:18,664 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.456742790465 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 79.9473350888 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 70.5028081486 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 58.4394373708 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.363333333333 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 00:33:18,687 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:33:23,759 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 4.94 2021-05-17 00:33:28,838 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 11.41 2021-05-17 00:33:33,914 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 17.72 2021-05-17 00:33:39,002 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 27.4 2021-05-17 00:33:44,525 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 00:33:49,608 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 42.8 2021-05-17 00:33:54,680 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 50.46 2021-05-17 00:33:59,749 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 62.46 2021-05-17 00:34:04,826 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 00:34:09,926 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 76.03 2021-05-17 00:34:15,013 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 81.03 2021-05-17 00:34:20,101 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 88.16 2021-05-17 00:34:25,250 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 95.36 2021-05-17 00:34:35,351 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:34:35,506 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 76.8420000076 sec 2021-05-17 00:34:43,563 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:35:30,158 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 00:35:32,105 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 00:35:33,043 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 00:35:33,045 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 00:35:45,290 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 00:35:45,292 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 00:36:04,190 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 00:36:04,191 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 00:37:02,770 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.135999918 seconds to complete 2021-05-17 00:37:02,790 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1130.905s During the test, OK Remote Connections: 172, Disconnections: 172 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 1 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_1 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 2, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:37:12,240 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:37:12,937 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:37:13,404 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff5 type=client open_count=1 channel=[id: 0xbc9b8c5c, 0.0.0.0/0.0.0.0:54272] timeout=300.0>) 2021-05-17 00:37:34,414 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff6 type=client open_count=1 channel=[id: 0x558e8f35, 0.0.0.0/0.0.0.0:52217] timeout=300.0>) 2021-05-17 00:37:36,434 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x99ff7 type=client open_count=1 channel=[id: 0x2072cf39, 0.0.0.0/0.0.0.0:45094] timeout=300.0>) 2021-05-17 00:37:38,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ff8 type=client open_count=1 channel=[id: 0x0dcede3e, 0.0.0.0/0.0.0.0:52625] timeout=300.0>) 2021-05-17 00:38:02,874 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ff9 type=client open_count=1 channel=[id: 0x33e45d13, 0.0.0.0/0.0.0.0:33161] timeout=300.0>) 2021-05-17 00:38:04,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x99ffa type=client open_count=1 channel=[id: 0x7bdac34b, 0.0.0.0/0.0.0.0:52880] timeout=300.0>) 2021-05-17 00:38:06,914 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffb type=client open_count=1 channel=[id: 0x9e140f7a, 0.0.0.0/0.0.0.0:56606] timeout=300.0>) 2021-05-17 00:38:30,625 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffc type=client open_count=1 channel=[id: 0xcdadd421, 0.0.0.0/0.0.0.0:47974] timeout=300.0>) 2021-05-17 00:38:32,645 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x99ffd type=client open_count=1 channel=[id: 0x98f07702, 0.0.0.0/0.0.0.0:39940] timeout=300.0>) 2021-05-17 00:38:34,664 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x99ffe type=client open_count=1 channel=[id: 0x6850fc93, 0.0.0.0/0.0.0.0:58029] timeout=300.0>) 2021-05-17 00:38:58,937 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x99fff type=client open_count=1 channel=[id: 0x6a2f6f47, 0.0.0.0/0.0.0.0:50821] timeout=300.0>) 2021-05-17 00:39:00,960 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x9a000 type=client open_count=1 channel=[id: 0x4af76def, 0.0.0.0/0.0.0.0:48388] timeout=300.0>) 2021-05-17 00:39:02,980 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a001 type=client open_count=1 channel=[id: 0x370fa448, 0.0.0.0/0.0.0.0:37162] timeout=300.0>) 2021-05-17 00:39:27,345 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a002 type=client open_count=1 channel=[id: 0xbc5635af, 0.0.0.0/0.0.0.0:45831] timeout=300.0>) 2021-05-17 00:39:29,364 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x9a003 type=client open_count=1 channel=[id: 0xbacf31e3, 0.0.0.0/0.0.0.0:43070] timeout=300.0>) 2021-05-17 00:39:31,384 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:39:48,543 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:40:02,201 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:40:02,226 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:40:02,266 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:40:16,194 | test | INFO | pool-4-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 0.365 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+-------+--------------+ 2021-05-17 00:40:16,220 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:40:26,296 | test | INFO | pool-4-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:40:26,323 | test | INFO | pool-4-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.129999876 sec 2021-05-17 00:40:26,578 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 00:41:14,964 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 00:41:15,384 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 865238056 | 120364342 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 291175888 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1126643576 | 83916269 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 00:41:15,387 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 00:41:58,657 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:41:58,670 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:42:19,477 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 169.413 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:42:20,063 | test | INFO | pool-4-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................................................**..................................................X | 26831.559 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:42:21,786 | test | INFO | pool-4-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 197.899 | | 92.226 | .............................................*X | 92.226 | +--------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 00:42:22,088 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.3200113331 | 6437847040 | 4303294464 | 2529480704 / 7511588864 | 82644 / 160843 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.5563973484 | 6437847040 | 4229492736 | 2583810048 / 7511588864 | 81474 / 246256 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.5309317931 | 6437847040 | 4256976896 | 2560475136 / 7511588864 | 80890 / 203264 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.532414803 | 6437847040 | 3915497472 | 3090124800 / 9530753024 | 82449 / 141853 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.3730564751 | 6437847040 | 4276359168 | 2538450944 / 7511588864 | 82311 / 208916 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 00:42:36,957 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:44:06,842 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212748328 | 298433305 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319770704 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 832897704 | 637498477 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 00:44:06,845 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #2 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 00:44:17,026 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 00:44:17,026 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 00:44:17,028 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 00:45:16,487 | test | INFO | pool-4-thread-27 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.2430000305 seconds to complete 2021-05-17 00:45:16,493 | test | INFO | pool-4-thread-27 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:47:21,559 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0620000362396 seconds 2021-05-17 00:48:12,648 | test | INFO | pool-4-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 50.9049999714 seconds to complete 2021-05-17 00:48:12,654 | test | INFO | pool-4-thread-26 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 00:50:17,717 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0590000152588 seconds 2021-05-17 00:50:17,719 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 00:50:17,743 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:50:24,059 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:50:46,012 | test | INFO | pool-4-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................**X | 3762.955 | +-------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:50:47,645 | test | INFO | pool-4-thread-9 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ****............................................................................***........................X | 17005.162 | +-------+--------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:50:49,187 | test | INFO | pool-4-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****................................*....................................................................*X | 2549.964 | | 2266.872 | .......................................................................................X | 2266.872 | | 1612.310 | ..............................................................X | 1612.310 | +----------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:50:49,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,582 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 00:50:49,582 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 00:51:03,928 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 00:51:04,059 | test | INFO | pool-4-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.624989583507 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 80.835814869 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 76.7170547158 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 55.3642560709 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.236662722288 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 00:51:04,088 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:51:09,177 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:51:10,825 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 00:51:14,584 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.76 2021-05-17 00:51:19,918 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.86 2021-05-17 00:51:25,168 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 1.67 2021-05-17 00:51:29,894 | test | INFO | pool-4-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........................................................................................**...........X | 2561.451 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:51:30,306 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 3.14 2021-05-17 00:51:31,786 | test | INFO | pool-4-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......................X | 5168.490 | | 5168.490 | ......................*...................................................*..........................X | 22099.988 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 00:51:33,701 | test | INFO | pool-4-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *...........*.......*..........................*........................................................*X | 4827.747 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 00:51:35,390 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 5.8 2021-05-17 00:51:40,476 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 9.01 2021-05-17 00:51:45,561 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.48 2021-05-17 00:51:50,647 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.21 2021-05-17 00:51:55,726 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 22.63 2021-05-17 00:52:00,819 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 28.49 2021-05-17 00:52:06,974 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 00:52:12,061 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.64 2021-05-17 00:52:17,150 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 48.01 2021-05-17 00:52:22,256 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 56.18 2021-05-17 00:52:27,342 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 64.64 2021-05-17 00:52:32,418 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.24 2021-05-17 00:52:37,502 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 76.09 2021-05-17 00:52:42,621 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 81.85 2021-05-17 00:52:47,727 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 87.82 2021-05-17 00:52:52,812 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 93.73 2021-05-17 00:52:57,888 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: running, progress: 99.39 2021-05-17 00:53:08,009 | test | INFO | pool-4-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:53:08,164 | test | INFO | pool-4-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 124.105999947 sec 2021-05-17 00:53:17,802 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 00:54:07,140 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 00:54:07,141 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 00:54:19,278 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 98% 2021-05-17 00:54:19,279 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 98% 2021-05-17 00:54:33,747 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 00:54:33,749 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 00:55:32,213 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1349999905 seconds to complete 2021-05-17 00:55:32,230 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1109.395s OK During the test, Remote Connections: 172, Disconnections: 172 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 2 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_2 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 3, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 00:55:41,701 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:55:42,367 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:55:42,798 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325c8 type=client open_count=1 channel=[id: 0x02d92d3c, 0.0.0.0/0.0.0.0:55453] timeout=300.0>) 2021-05-17 00:56:03,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325c9 type=client open_count=1 channel=[id: 0x16a1c393, 0.0.0.0/0.0.0.0:41002] timeout=300.0>) 2021-05-17 00:56:05,798 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325ca type=client open_count=1 channel=[id: 0xbb1a638d, 0.0.0.0/0.0.0.0:57178] timeout=300.0>) 2021-05-17 00:56:07,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325cb type=client open_count=1 channel=[id: 0xcdb74187, 0.0.0.0/0.0.0.0:46543] timeout=300.0>) 2021-05-17 00:56:09,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1325cc type=client open_count=1 channel=[id: 0x4e9edf2e, 0.0.0.0/0.0.0.0:39153] timeout=300.0>) 2021-05-17 00:56:11,857 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325cd type=client open_count=1 channel=[id: 0x7d0308a4, 0.0.0.0/0.0.0.0:46836] timeout=300.0>) 2021-05-17 00:56:38,328 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325ce type=client open_count=1 channel=[id: 0x194d6629, 0.0.0.0/0.0.0.0:51793] timeout=300.0>) 2021-05-17 00:56:40,346 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325cf type=client open_count=1 channel=[id: 0x6f242d88, 0.0.0.0/0.0.0.0:55222] timeout=300.0>) 2021-05-17 00:56:42,368 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1325d0 type=client open_count=1 channel=[id: 0x9b3aaa75, 0.0.0.0/0.0.0.0:48828] timeout=300.0>) 2021-05-17 00:56:44,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d1 type=client open_count=1 channel=[id: 0x11dca817, 0.0.0.0/0.0.0.0:37305] timeout=300.0>) 2021-05-17 00:57:08,474 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d2 type=client open_count=1 channel=[id: 0x5964fad3, 0.0.0.0/0.0.0.0:43020] timeout=300.0>) 2021-05-17 00:57:10,493 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d3 type=client open_count=1 channel=[id: 0x99032c2a, 0.0.0.0/0.0.0.0:33179] timeout=300.0>) 2021-05-17 00:57:12,512 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1325d4 type=client open_count=1 channel=[id: 0x8edf7874, 0.0.0.0/0.0.0.0:53068] timeout=300.0>) 2021-05-17 00:57:14,533 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d5 type=client open_count=1 channel=[id: 0x357232ce, 0.0.0.0/0.0.0.0:34306] timeout=300.0>) 2021-05-17 00:57:38,243 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d6 type=client open_count=1 channel=[id: 0xa273d954, 0.0.0.0/0.0.0.0:37356] timeout=300.0>) 2021-05-17 00:57:40,262 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1325d7 type=client open_count=1 channel=[id: 0x3828793b, 0.0.0.0/0.0.0.0:36633] timeout=300.0>) 2021-05-17 00:57:42,280 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325d8 type=client open_count=1 channel=[id: 0x84e3337b, 0.0.0.0/0.0.0.0:57379] timeout=300.0>) 2021-05-17 00:58:06,551 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325d9 type=client open_count=1 channel=[id: 0xbcae4a5c, 0.0.0.0/0.0.0.0:48979] timeout=300.0>) 2021-05-17 00:58:08,569 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325da type=client open_count=1 channel=[id: 0x0e1b016f, 0.0.0.0/0.0.0.0:43883] timeout=300.0>) 2021-05-17 00:58:10,588 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325db type=client open_count=1 channel=[id: 0x8690fc52, 0.0.0.0/0.0.0.0:53603] timeout=300.0>) 2021-05-17 00:58:12,607 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325dc type=client open_count=1 channel=[id: 0xbb1605a2, 0.0.0.0/0.0.0.0:55398] timeout=300.0>) 2021-05-17 00:58:14,627 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1325dd type=client open_count=1 channel=[id: 0xfd39469e, 0.0.0.0/0.0.0.0:43569] timeout=300.0>) 2021-05-17 00:58:16,661 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 00:58:33,713 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 00:58:47,446 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 00:58:47,467 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:58:47,510 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 00:59:00,839 | test | INFO | pool-5-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.51660805653 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 00:59:00,875 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 00:59:10,950 | test | INFO | pool-5-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 00:59:10,976 | test | INFO | pool-5-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1380000114 sec 2021-05-17 00:59:11,236 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:00:10,711 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:00:10,982 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 774107752 | 157838792 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 290193472 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 868677800 | 111247271 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:00:10,983 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:00:55,142 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:00:55,154 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:01:17,453 | test | INFO | pool-5-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 129.747 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:01:17,707 | test | INFO | pool-5-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....................................................*...............................................*X | 108.356 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:01:19,117 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..................................................................................................**..X | 23952.315 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:01:19,407 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.3540586647 | 6437847040 | 4138688512 | 2606309376 / 7511588864 | 96935 / 293317 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.846435825 | 6437847040 | 4145115136 | 2598170624 / 7511588864 | 95660 / 226680 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9198784824 | 6437847040 | 4110794752 | 2603536384 / 7511588864 | 94704 / 253010 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.8948369973 | 6437847040 | 3740631040 | 3179642880 / 9530753024 | 96785 / 286707 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8946682048 | 6437847040 | 4256780288 | 2567884800 / 7511588864 | 95533 / 242609 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:01:33,538 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:03:00,661 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210910616 | 324906015 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319777712 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 814533768 | 627370453 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:03:00,663 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #3 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:03:11,013 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 01:03:12,667 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 01:03:12,668 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_rebalance_out 2021-05-17 01:03:12,670 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:03:23,164 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:03:23,381 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 01:03:23,381 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:03:37,448 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2409 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2467 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2434 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2405 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2416 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2462 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2431 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2422 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2400 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:37,450 | test | INFO | pool-5-thread-11 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2409 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2434 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2467 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2405 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2416 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2462 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2431 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-2422 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-2400 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:38,805 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 9 docs from 0 to 5000 2021-05-17 01:03:41,943 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4192 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4181 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4184 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4151 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4173 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4160 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4129 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4166 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4123 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4175 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4142 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4011 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4158 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4169 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4136 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:41,946 | test | INFO | pool-5-thread-30 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4192 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4181 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4184 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4151 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4173 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4160 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4129 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4166 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4123 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4142 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4175 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4011 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS] | | test_collections-4158 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4169 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4136 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,585 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4708 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4735 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4757 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4758 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4712 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4723 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4717 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4739 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4726 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4682 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4770 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4687 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4775 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4698 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4797 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4798 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4666 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4763 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4730 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,588 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4708 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4757 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4735 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4758 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4723 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4712 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4717 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4739 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4726 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4682 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4770 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4687 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4698 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4797 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4775 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4666 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4798 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4730 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4763 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:03:42,625 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 19 docs from 0 to 5000 2021-05-17 01:03:42,628 | test | ERROR | pool-5-thread-4 | [task:call:3809] Failed to load 15 docs from 0 to 5000 2021-05-17 01:03:44,098 | test | INFO | pool-5-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5722.713 | | 1468.823 | ........................X | 1468.823 | | 59.945 | X | 59.945 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:03:44,141 | test | INFO | pool-5-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 2626.305 | | 1917.360 | ........................................................................X | 1917.360 | | 586.883 | .....................X | 586.883 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:03:45,553 | test | INFO | pool-5-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.......................................................................................*.............X | 21005.938 | | 20847.741 | ..................................................................................................X | 20847.741 | | 20779.914 | .................................................................................................X | 20779.914 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:03:57,592 | test | INFO | pool-5-thread-9 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.20700001717 seconds to complete 2021-05-17 01:03:57,599 | test | INFO | pool-5-thread-9 | [common_lib:sleep:22] Sleep 8.20700001717 seconds. Reason: Wait after rebalance complete 2021-05-17 01:06:05,869 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0589997768402 seconds 2021-05-17 01:06:17,476 | test | INFO | pool-5-thread-6 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.48200011253 seconds to complete 2021-05-17 01:06:17,482 | test | INFO | pool-5-thread-6 | [common_lib:sleep:22] Sleep 8.48200011253 seconds. Reason: Wait after rebalance complete 2021-05-17 01:08:26,022 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 01:08:26,023 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:08:26,023 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:08:33,029 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:08:36,440 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,512 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,622 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,684 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,954 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:36,954 | test | INFO | pool-5-thread-17 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,029 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,115 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,216 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:37,407 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,459 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:38,464 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:38,700 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,760 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1292 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1202 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1225 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1247 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1302 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1369 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1248 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1347 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1207 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1229 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1329 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1260 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1384 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1220 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1242 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1287 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1265 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1288 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1388 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1281 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1214 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1236 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1358 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1218 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1317 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1318 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1392 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1273 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1251 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1352 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1276 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1397 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1254 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1233 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1376 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1354 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1311 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1333 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,877 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1408 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1529 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1544 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1566 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1501 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1523 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1404 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1448 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1426 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1504 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1548 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1526 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1482 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1441 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1463 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1563 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1420 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1541 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1465 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1487 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1598 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1435 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1512 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1479 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1557 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1535 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1579 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1439 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1517 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1491 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1470 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1570 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1494 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1594 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1552 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1530 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1575 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:38,969 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,134 | test | INFO | pool-5-thread-19 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,263 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1090 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1071 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1049 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1005 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1027 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1127 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1105 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1009 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1062 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1040 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1162 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1140 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1064 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1086 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1021 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1122 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1100 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1167 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1145 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1083 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1038 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1116 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1095 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1195 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1053 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1174 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1153 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1032 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1131 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1077 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1034 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1199 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1078 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1134 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1156 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1178 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1113 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,493 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1628 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1705 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1606 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1728 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1686 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1785 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1664 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1621 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1721 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1689 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1789 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1624 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1768 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1603 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1746 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1703 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1649 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1680 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1780 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1661 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1740 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1716 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1619 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1719 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1796 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1610 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1632 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1753 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1655 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1732 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1710 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1677 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1777 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1755 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1637 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1615 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1759 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1693 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1650 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1672 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1793 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1695 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:39,572 | test | INFO | pool-5-thread-15 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1947 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1925 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1829 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1884 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1920 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1942 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1825 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1902 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1869 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1881 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1860 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1914 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1936 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1818 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1939 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1851 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1873 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1951 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1830 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1875 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1954 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1933 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1911 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1836 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1814 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1858 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1892 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,002 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,282 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,426 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,434 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,489 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,497 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,578 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,663 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,697 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:idgwDTv-22-62000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:41,773 | test | INFO | pool-5-thread-6 | [table_view:display:72] Keys failed in default:JT%Dnv0BrAuNpM1fir0d-22-645000:-R-22-751000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,512 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 188 docs from 1000 to 1960 2021-05-17 01:08:42,516 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 111 docs from 1000 to 1960 2021-05-17 01:08:42,642 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,730 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,835 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,888 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:42,993 | test | INFO | pool-5-thread-26 | [table_view:display:72] Keys failed in default:A1WmWR9uKz1rXS--21-905000:ocHTA%6xh5N_a1D-22-63000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:43,022 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-16 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:43,147 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:44,976 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-508 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-407 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-429 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-507 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-502 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-469 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-547 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-525 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-569 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-588 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-401 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-520 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-542 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-444 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-565 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-584 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-442 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-560 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-582 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-484 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-481 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-418 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-436 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-458 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-414 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-536 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-558 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-514 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-511 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-533 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-457 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-597 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-498 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-576 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-554 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-551 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-573 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-473 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-451 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-492 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-591 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:45,023 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-709 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-605 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-627 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-749 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-724 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-667 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-645 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-722 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-700 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-745 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-767 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-668 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-786 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-600 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-622 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-761 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-662 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-640 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-783 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-616 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-738 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-634 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-656 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-799 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-778 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-613 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-731 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-699 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-674 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-795 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-653 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-774 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-631 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-671 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-690 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-790 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:45,086 | test | INFO | pool-5-thread-23 | [table_view:display:72] Keys failed in bucket2:ZRPsBUtqo4MwvYCKcUs-20-407000:RTU-20-543000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-808 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-904 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-926 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-948 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-847 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-804 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-826 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-848 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-944 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-966 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-923 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-901 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-887 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-888 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-987 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-863 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-841 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-941 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-882 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-981 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-917 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-918 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-817 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-839 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-935 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-979 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-957 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-912 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-879 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-975 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-854 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-811 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-998 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-852 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-952 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-930 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-894 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-994 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-870 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-970 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-891 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 01:08:46,233 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,243 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,253 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:46,260 | test | ERROR | pool-5-thread-21 | [task:call:3809] Failed to load 203 docs from 0 to 1000 2021-05-17 01:08:49,492 | test | INFO | pool-5-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..............*......................................................................................X | 3960.129 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:08:49,572 | test | INFO | pool-5-thread-14 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*...........................................................................................X | 4115.282 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:08:49,740 | test | INFO | pool-5-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................*...............................................................................X | 15759.419 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:08:49,832 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,834 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,894 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:08:49,894 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:09:07,670 | test | INFO | pool-5-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.726690889696 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 73.8 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 85 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 62.6041736116 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.73 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:09:07,693 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:09:12,776 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.41 2021-05-17 01:09:17,887 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 8.99 2021-05-17 01:09:22,973 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 17.09 2021-05-17 01:09:28,058 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 27.51 2021-05-17 01:09:33,365 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 01:09:38,446 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 44.99 2021-05-17 01:09:43,601 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.59 2021-05-17 01:09:48,673 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:09:53,737 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.29 2021-05-17 01:09:58,826 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 80.84 2021-05-17 01:10:03,900 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 88.53 2021-05-17 01:10:08,967 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 95.46 2021-05-17 01:10:14,094 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 01:10:24,184 | test | INFO | pool-5-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:10:24,342 | test | INFO | pool-5-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 76.6729998589 sec 2021-05-17 01:10:24,344 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 01:10:26,309 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 01:10:27,276 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:10:27,276 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:10:39,993 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:10:39,993 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:10:55,023 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:10:55,025 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:11:51,622 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.135999918 seconds to complete 2021-05-17 01:11:51,638 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 979.373s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 3 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_3 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 4, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:12:01,002 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:12:01,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:12:02,122 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dbe type=client open_count=1 channel=[id: 0x44871791, 0.0.0.0/0.0.0.0:48925] timeout=300.0>) 2021-05-17 01:12:22,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dbf type=client open_count=1 channel=[id: 0x74a7c330, 0.0.0.0/0.0.0.0:51529] timeout=300.0>) 2021-05-17 01:12:24,875 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x1c6dc0 type=client open_count=1 channel=[id: 0xf1a2d509, 0.0.0.0/0.0.0.0:57167] timeout=300.0>) 2021-05-17 01:12:26,894 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc1 type=client open_count=1 channel=[id: 0x7264de9a, 0.0.0.0/0.0.0.0:43414] timeout=300.0>) 2021-05-17 01:12:51,380 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc2 type=client open_count=1 channel=[id: 0x0ad626fa, 0.0.0.0/0.0.0.0:35832] timeout=300.0>) 2021-05-17 01:12:53,404 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x1c6dc3 type=client open_count=1 channel=[id: 0xb30e846f, 0.0.0.0/0.0.0.0:50787] timeout=300.0>) 2021-05-17 01:12:55,421 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc4 type=client open_count=1 channel=[id: 0xbfa8bd60, 0.0.0.0/0.0.0.0:45145] timeout=300.0>) 2021-05-17 01:13:19,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc5 type=client open_count=1 channel=[id: 0xb5f72826, 0.0.0.0/0.0.0.0:37941] timeout=300.0>) 2021-05-17 01:13:21,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x1c6dc6 type=client open_count=1 channel=[id: 0x9b840ffc, 0.0.0.0/0.0.0.0:41204] timeout=300.0>) 2021-05-17 01:13:23,835 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc7 type=client open_count=1 channel=[id: 0x311f6ce4, 0.0.0.0/0.0.0.0:59078] timeout=300.0>) 2021-05-17 01:13:49,127 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc8 type=client open_count=1 channel=[id: 0x819cb21d, 0.0.0.0/0.0.0.0:60464] timeout=300.0>) 2021-05-17 01:13:51,145 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dc9 type=client open_count=1 channel=[id: 0x386f1fc1, 0.0.0.0/0.0.0.0:47323] timeout=300.0>) 2021-05-17 01:13:53,164 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dca type=client open_count=1 channel=[id: 0x8041e755, 0.0.0.0/0.0.0.0:50512] timeout=300.0>) 2021-05-17 01:13:55,181 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dcb type=client open_count=1 channel=[id: 0x8f2b126a, 0.0.0.0/0.0.0.0:46574] timeout=300.0>) 2021-05-17 01:13:57,200 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x1c6dcc type=client open_count=1 channel=[id: 0x3a291f1f, 0.0.0.0/0.0.0.0:53683] timeout=300.0>) 2021-05-17 01:13:59,217 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dcd type=client open_count=1 channel=[id: 0xf0058cf4, 0.0.0.0/0.0.0.0:40129] timeout=300.0>) 2021-05-17 01:14:23,489 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dce type=client open_count=1 channel=[id: 0xfec4ed39, 0.0.0.0/0.0.0.0:60225] timeout=300.0>) 2021-05-17 01:14:25,506 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x1c6dcf type=client open_count=1 channel=[id: 0x25e895d6, 0.0.0.0/0.0.0.0:59273] timeout=300.0>) 2021-05-17 01:14:27,525 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:14:44,849 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:14:58,641 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:14:58,664 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:14:58,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:15:12,809 | test | INFO | pool-6-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.00316656111 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:15:12,835 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:15:22,905 | test | INFO | pool-6-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:15:22,931 | test | INFO | pool-6-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1210000515 sec 2021-05-17 01:15:23,183 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:16:11,423 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:16:11,854 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 825103016 | 138629819 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 292133600 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1210977192 | 61923801 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 01:16:11,855 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:16:54,559 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:16:54,571 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:17:16,653 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 197.169 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:17:18,507 | test | INFO | pool-6-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..................................................................................**..................X | 159.482 | +-------+------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:17:19,947 | test | INFO | pool-6-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.**...................................................................................................*X | 24184.038 | | 24162.719 | ..................................................................................................X | 24162.719 | | 23925.579 | .................................................................................................X | 23925.579 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:17:20,214 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.2454845925 | 6437847040 | 4030156800 | 2746650624 / 7511588864 | 51020 / 150232 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.6373048453 | 6437847040 | 4196646912 | 2590498816 / 7511588864 | 93610 / 284097 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9466693332 | 6437847040 | 4194025472 | 2584666112 / 7511588864 | 100400 / 240179 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.3244030285 | 6437847040 | 3812818944 | 3177824256 / 9530753024 | 99128 / 223527 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.6377955372 | 6437847040 | 4154380288 | 2567847936 / 7511588864 | 98305 / 266741 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:17:34,286 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:19:01,371 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211369736 | 314354858 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320191280 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 828010104 | 609052213 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:19:01,371 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #4 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 01:19:11,596 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 01:19:11,598 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 01:19:11,598 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:19:22,786 | test | INFO | pool-6-thread-19 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.23000001907 seconds to complete 2021-05-17 01:19:22,790 | test | INFO | pool-6-thread-19 | [common_lib:sleep:22] Sleep 8.23000001907 seconds. Reason: Wait after rebalance complete 2021-05-17 01:21:31,082 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0579998493195 seconds 2021-05-17 01:21:42,753 | test | INFO | pool-6-thread-5 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.16799998283 seconds to complete 2021-05-17 01:21:42,757 | test | INFO | pool-6-thread-5 | [common_lib:sleep:22] Sleep 8.16799998283 seconds. Reason: Wait after rebalance complete 2021-05-17 01:23:50,986 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0570001602173 seconds 2021-05-17 01:23:50,986 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:23:50,987 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:23:58,273 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:24:13,986 | test | INFO | pool-6-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..............................................*......................................................X | 1161.150 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:24:14,822 | test | INFO | pool-6-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 1604.391 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:24:14,861 | test | INFO | pool-6-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................*X | 13291.347 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:24:14,974 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:24:14,976 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:24:15,042 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:24:15,042 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:24:33,118 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:24:33,346 | test | INFO | pool-6-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.521657972367 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 72.7189263179 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 70.8749226731 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 64.765 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.548324194597 | --- OUT ---> | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:24:33,398 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:24:38,625 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:24:40,936 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:24:43,845 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.92 2021-05-17 01:24:49,157 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.92 2021-05-17 01:24:55,318 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 2.78 2021-05-17 01:25:00,247 | test | INFO | pool-6-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *........................................................*............................................*X | 2088.676 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:25:00,523 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 3.76 2021-05-17 01:25:01,934 | test | INFO | pool-6-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.........*........**...................................................................................X | 3378.401 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:25:02,246 | test | INFO | pool-6-thread-19 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................................X | 5642.696 | | 5642.696 | ....................................*...............................................................*X | 15099.033 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:25:05,621 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 6.53 2021-05-17 01:25:10,700 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 9.36 2021-05-17 01:25:15,798 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.61 2021-05-17 01:25:20,914 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.93 2021-05-17 01:25:26,065 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.11 2021-05-17 01:25:31,151 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.39 2021-05-17 01:25:36,232 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.34 2021-05-17 01:25:41,322 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 44.26 2021-05-17 01:25:46,433 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 51.73 2021-05-17 01:25:51,529 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 59.88 2021-05-17 01:25:56,619 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:26:01,687 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 71.94 2021-05-17 01:26:06,779 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 78.08 2021-05-17 01:26:11,859 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 84.53 2021-05-17 01:26:16,941 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 91.04 2021-05-17 01:26:22,023 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: running, progress: 97.43 2021-05-17 01:26:32,137 | test | INFO | pool-6-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:26:32,286 | test | INFO | pool-6-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 118.940999985 sec 2021-05-17 01:26:33,292 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:26:33,292 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:26:46,046 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:26:46,048 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:27:05,269 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:27:05,269 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:28:00,411 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-17 01:28:00,427 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 968.763s During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 4 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_4 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 5, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:28:09,954 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:28:10,617 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:28:11,043 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f0 type=client open_count=1 channel=[id: 0x07d297e3, 0.0.0.0/0.0.0.0:60180] timeout=300.0>) 2021-05-17 01:28:31,861 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f1 type=client open_count=1 channel=[id: 0xc8ac43f2, 0.0.0.0/0.0.0.0:59511] timeout=300.0>) 2021-05-17 01:28:33,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x25f7f2 type=client open_count=1 channel=[id: 0x90162b89, 0.0.0.0/0.0.0.0:55137] timeout=300.0>) 2021-05-17 01:28:35,897 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f3 type=client open_count=1 channel=[id: 0x2afb44ac, 0.0.0.0/0.0.0.0:41323] timeout=300.0>) 2021-05-17 01:28:59,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f4 type=client open_count=1 channel=[id: 0x6e49327e, 0.0.0.0/0.0.0.0:60719] timeout=300.0>) 2021-05-17 01:29:01,799 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x25f7f5 type=client open_count=1 channel=[id: 0x9ad59d96, 0.0.0.0/0.0.0.0:42588] timeout=300.0>) 2021-05-17 01:29:03,818 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f6 type=client open_count=1 channel=[id: 0x56d3defa, 0.0.0.0/0.0.0.0:33068] timeout=300.0>) 2021-05-17 01:29:27,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f7 type=client open_count=1 channel=[id: 0x234f468f, 0.0.0.0/0.0.0.0:55615] timeout=300.0>) 2021-05-17 01:29:29,862 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x25f7f8 type=client open_count=1 channel=[id: 0x6f822940, 0.0.0.0/0.0.0.0:48443] timeout=300.0>) 2021-05-17 01:29:31,881 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7f9 type=client open_count=1 channel=[id: 0x694fa3c8, 0.0.0.0/0.0.0.0:57900] timeout=300.0>) 2021-05-17 01:29:56,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7fa type=client open_count=1 channel=[id: 0x62443c46, 0.0.0.0/0.0.0.0:50138] timeout=300.0>) 2021-05-17 01:29:58,599 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x25f7fb type=client open_count=1 channel=[id: 0x48af4212, 0.0.0.0/0.0.0.0:41224] timeout=300.0>) 2021-05-17 01:30:00,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fc type=client open_count=1 channel=[id: 0xa47f004d, 0.0.0.0/0.0.0.0:59331] timeout=300.0>) 2021-05-17 01:30:24,759 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fd type=client open_count=1 channel=[id: 0xf1891a05, 0.0.0.0/0.0.0.0:40322] timeout=300.0>) 2021-05-17 01:30:26,776 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x25f7fe type=client open_count=1 channel=[id: 0xe6c84fd4, 0.0.0.0/0.0.0.0:37271] timeout=300.0>) 2021-05-17 01:30:28,795 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:30:46,052 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:30:59,707 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:30:59,729 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:30:59,766 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:31:13,875 | test | INFO | pool-7-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.66158897352 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:31:13,901 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:31:23,980 | test | INFO | pool-7-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:31:24,007 | test | INFO | pool-7-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1339998245 sec 2021-05-17 01:31:24,273 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:32:16,796 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:32:17,151 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 876682040 | 135632571 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 297069504 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 929359320 | 107314098 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:32:17,151 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:33:01,072 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:33:01,085 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:33:23,234 | test | INFO | pool-7-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.................................**...................................................................*X | 24735.895 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:33:24,733 | test | INFO | pool-7-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.....***...............................................................................................*X | 146.677 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:33:25,388 | test | INFO | pool-7-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **...........................**.........................................................................**X | 167.644 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:33:25,654 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.6606202458 | 6437847040 | 4158156800 | 2617679872 / 7511588864 | 67670 / 198927 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 98.4383073051 | 6437847040 | 4208533504 | 2569297920 / 7511588864 | 67116 / 81341 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.7404066962 | 6437847040 | 4152954880 | 2610405376 / 7511588864 | 66092 / 170574 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.973326665 | 6437847040 | 3609845760 | 3354857472 / 9530753024 | 101800 / 287891 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.8452424535 | 6437847040 | 4181737472 | 2615734272 / 7511588864 | 66794 / 168941 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:33:40,891 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:35:08,010 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210825048 | 318226215 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320723760 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 833004360 | 613302031 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:35:08,012 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #5 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:35:18,207 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 01:35:19,835 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 01:35:19,835 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_recovery 2021-05-17 01:35:19,836 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:35:29,049 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:35:29,226 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 01:35:29,226 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:35:44,930 | test | INFO | pool-7-thread-24 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...................................................................................................X | 11509.229 | +-------+-------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:35:47,904 | test | INFO | pool-7-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5006.430 | | 106.522 | .X | 106.522 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:35:48,321 | test | INFO | pool-7-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5250.357 | | 530.773 | .........X | 530.773 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:36:56,190 | test | INFO | pool-7-thread-14 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 86.6129999161 seconds to complete 2021-05-17 01:36:56,197 | test | INFO | pool-7-thread-14 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:39:01,257 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 01:39:46,903 | test | INFO | pool-7-thread-23 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 45.4580001831 seconds to complete 2021-05-17 01:39:46,907 | test | INFO | pool-7-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:41:51,967 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 01:41:52,161 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 01:41:52,164 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:41:59,158 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:42:12,201 | test | INFO | pool-7-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 18262.328 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:42:12,394 | test | INFO | pool-7-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 8652.624 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:42:12,482 | test | INFO | pool-7-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5789.830 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 01:42:12,680 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,680 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,732 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 01:42:12,732 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 01:42:29,418 | test | INFO | pool-7-thread-28 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.416680556019 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 81.0947761443 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 66.98110063 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 72.8124270809 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.679977334089 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 01:42:29,440 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:34,497 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:39,563 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:42:44,746 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.31 2021-05-17 01:42:49,930 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.79 2021-05-17 01:42:55,204 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 14.52 2021-05-17 01:43:00,500 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 19.94 2021-05-17 01:43:06,013 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 22.89 2021-05-17 01:43:11,463 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 28.1 2021-05-17 01:43:16,742 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 01:43:23,822 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.29 2021-05-17 01:43:29,588 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 53.13 2021-05-17 01:43:34,723 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 58.4 2021-05-17 01:43:40,063 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 01:43:45,171 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 66.81 2021-05-17 01:43:50,967 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 75.28 2021-05-17 01:43:56,055 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 79.61 2021-05-17 01:44:01,138 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 83.04 2021-05-17 01:44:06,209 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 85.91 2021-05-17 01:44:11,329 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 90.95 2021-05-17 01:44:16,414 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 92.3 2021-05-17 01:44:21,653 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: running, progress: 98.46 2021-05-17 01:44:31,743 | test | INFO | pool-7-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:44:31,783 | test | INFO | pool-7-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 122.365999937 sec 2021-05-17 01:44:44,157 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:45:58,720 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 01:46:00,719 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 01:46:01,674 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 01:46:01,676 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 01:46:14,240 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 01:46:14,240 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 01:46:35,573 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 01:46:35,575 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 01:47:30,551 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 01:47:30,565 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 Ran 1 test in 1170.185s OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 5 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_5 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 6, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 01:47:40,234 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:47:41,035 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:47:41,457 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e5f type=client open_count=1 channel=[id: 0x8e9bc38e, 0.0.0.0/0.0.0.0:50976] timeout=300.0>) 2021-05-17 01:48:02,299 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e60 type=client open_count=1 channel=[id: 0x3e3b91f8, 0.0.0.0/0.0.0.0:39510] timeout=300.0>) 2021-05-17 01:48:04,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x2f4e61 type=client open_count=1 channel=[id: 0xb2b39248, 0.0.0.0/0.0.0.0:57881] timeout=300.0>) 2021-05-17 01:48:06,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e62 type=client open_count=1 channel=[id: 0x72e81b02, 0.0.0.0/0.0.0.0:59983] timeout=300.0>) 2021-05-17 01:48:30,150 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e63 type=client open_count=1 channel=[id: 0xb46d55e3, 0.0.0.0/0.0.0.0:34739] timeout=300.0>) 2021-05-17 01:48:32,168 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x2f4e64 type=client open_count=1 channel=[id: 0x68b8f4a9, 0.0.0.0/0.0.0.0:57525] timeout=300.0>) 2021-05-17 01:48:34,186 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e65 type=client open_count=1 channel=[id: 0x456eb8d6, 0.0.0.0/0.0.0.0:58685] timeout=300.0>) 2021-05-17 01:48:58,158 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e66 type=client open_count=1 channel=[id: 0x2beb4074, 0.0.0.0/0.0.0.0:34432] timeout=300.0>) 2021-05-17 01:49:00,177 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x2f4e67 type=client open_count=1 channel=[id: 0xff5e2815, 0.0.0.0/0.0.0.0:50019] timeout=300.0>) 2021-05-17 01:49:02,194 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e68 type=client open_count=1 channel=[id: 0x0ab5470c, 0.0.0.0/0.0.0.0:35960] timeout=300.0>) 2021-05-17 01:49:26,186 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e69 type=client open_count=1 channel=[id: 0x2c8c1613, 0.0.0.0/0.0.0.0:60328] timeout=300.0>) 2021-05-17 01:49:28,204 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x2f4e6a type=client open_count=1 channel=[id: 0x0cc5d59d, 0.0.0.0/0.0.0.0:42156] timeout=300.0>) 2021-05-17 01:49:30,223 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6b type=client open_count=1 channel=[id: 0x5c73f1e5, 0.0.0.0/0.0.0.0:49283] timeout=300.0>) 2021-05-17 01:49:54,253 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6c type=client open_count=1 channel=[id: 0x40f4876a, 0.0.0.0/0.0.0.0:34894] timeout=300.0>) 2021-05-17 01:49:56,269 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x2f4e6d type=client open_count=1 channel=[id: 0xf0c30c92, 0.0.0.0/0.0.0.0:43172] timeout=300.0>) 2021-05-17 01:49:58,288 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 01:50:15,482 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 01:50:29,153 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 01:50:29,176 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:50:29,213 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:50:43,017 | test | INFO | pool-8-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 11.5898068366 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 01:50:43,042 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 01:50:53,115 | test | INFO | pool-8-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 01:50:53,141 | test | INFO | pool-8-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.125 sec 2021-05-17 01:50:53,394 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 01:51:46,911 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 01:51:47,207 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 780461256 | 150646608 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 287649472 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 958313032 | 93412038 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 01:51:47,207 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 01:52:29,881 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 01:52:29,894 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 01:52:53,505 | test | INFO | pool-8-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.......................................*...*............................*..............................X | 167.652 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:52:53,561 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..................**.......................................................*...........................X | 176.450 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 01:52:54,325 | test | INFO | pool-8-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *......................**..............................................................................*X | 26582.246 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 01:52:54,614 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 100 | 6437847040 | 4136271872 | 2635771904 / 7511588864 | 66721 / 199077 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 81.4132385193 | 6437847040 | 4243619840 | 2539024384 / 7511588864 | 64700 / 150690 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 84.5816441816 | 6437847040 | 4263325696 | 2506563584 / 7511588864 | 66317 / 197277 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 83.7487935568 | 6437847040 | 3862028288 | 3106611200 / 9530753024 | 66597 / 175073 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 82.3222017618 | 6437847040 | 4297330688 | 2500845568 / 7511588864 | 66070 / 164061 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 01:53:09,421 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 01:54:36,729 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211783656 | 312935045 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319237296 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 842485320 | 619182210 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 01:54:36,730 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #6 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 01:54:46,914 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 01:55:46,822 | test | INFO | pool-8-thread-16 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.7079999447 seconds to complete 2021-05-17 01:55:46,828 | test | INFO | pool-8-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 01:57:51,888 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 01:58:34,700 | test | INFO | pool-8-thread-8 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 42.631000042 seconds to complete 2021-05-17 01:58:34,710 | test | INFO | pool-8-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:00:39,828 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.115000009537 seconds 2021-05-17 02:00:39,828 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:00:39,829 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:00:47,345 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:01:05,858 | test | INFO | pool-8-thread-2 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 2231.392 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:06,885 | test | INFO | pool-8-thread-24 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***...................................................................................................**X | 3157.434 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:07,720 | test | INFO | pool-8-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................*.......................................**...........................................X | 20446.615 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:01:08,019 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,020 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,148 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:01:08,150 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:01:22,829 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:01:23,105 | test | INFO | pool-8-thread-8 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.546666666667 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 84.2980716345 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 83.1499874802 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 59.2716666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.576657055716 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:01:23,267 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:28,510 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:32,266 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:01:33,677 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:38,905 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:44,142 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:50,470 | test | INFO | pool-8-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ****....................................................................................................X | 5134.506 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:51,552 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:01:52,575 | test | INFO | pool-8-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........***..........................................................................................*X | 4447.726 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:01:53,000 | test | INFO | pool-8-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-------+------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...........*......**...................................................................................*X | 18994.644 | +-------+------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:01:56,690 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 4.23 2021-05-17 02:02:01,888 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 7.29 2021-05-17 02:02:07,056 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 10.67 2021-05-17 02:02:12,229 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 13.98 2021-05-17 02:02:17,359 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 17.69 2021-05-17 02:02:22,499 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.73 2021-05-17 02:02:27,591 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 25.6 2021-05-17 02:02:32,757 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 29.12 2021-05-17 02:02:37,865 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 32.13 2021-05-17 02:02:44,581 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 34.32 2021-05-17 02:02:49,780 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 43.81 2021-05-17 02:02:55,055 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 50.71 2021-05-17 02:03:00,480 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 57.54 2021-05-17 02:03:05,789 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 63.23 2021-05-17 02:03:10,950 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 02:03:16,040 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.16 2021-05-17 02:03:21,240 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 75.64 2021-05-17 02:03:26,433 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.87 2021-05-17 02:03:31,585 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 83.51 2021-05-17 02:03:36,770 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 86.35 2021-05-17 02:03:42,009 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 90.13 2021-05-17 02:03:47,247 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 93.55 2021-05-17 02:03:52,451 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 96.17 2021-05-17 02:03:57,605 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: running, progress: 98.75 2021-05-17 02:04:07,838 | test | INFO | pool-8-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:04:07,934 | test | INFO | pool-8-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 164.828999996 sec 2021-05-17 02:04:34,028 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:06:23,842 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:06:23,844 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:06:42,855 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 02:06:42,855 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 02:07:06,572 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:07:06,572 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:08:12,371 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.2690000534 seconds to complete 2021-05-17 02:08:12,387 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1242.445s OK During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 6 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_6 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 7, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:08:27,801 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:08:29,414 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:08:31,522 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a744 type=client open_count=1 channel=[id: 0x29521add, 0.0.0.0/0.0.0.0:45172] timeout=300.0>) 2021-05-17 02:08:56,769 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a745 type=client open_count=1 channel=[id: 0x5c81ece5, 0.0.0.0/0.0.0.0:40790] timeout=300.0>) 2021-05-17 02:08:58,821 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x38a746 type=client open_count=1 channel=[id: 0x01dd2be8, 0.0.0.0/0.0.0.0:56805] timeout=300.0>) 2021-05-17 02:09:00,865 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a747 type=client open_count=1 channel=[id: 0xf61703d9, 0.0.0.0/0.0.0.0:34973] timeout=300.0>) 2021-05-17 02:09:26,427 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a748 type=client open_count=1 channel=[id: 0xd5a93556, 0.0.0.0/0.0.0.0:52391] timeout=300.0>) 2021-05-17 02:09:28,453 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x38a749 type=client open_count=1 channel=[id: 0xc019a27b, 0.0.0.0/0.0.0.0:50454] timeout=300.0>) 2021-05-17 02:09:30,479 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74a type=client open_count=1 channel=[id: 0x8ca75604, 0.0.0.0/0.0.0.0:39009] timeout=300.0>) 2021-05-17 02:09:55,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74b type=client open_count=1 channel=[id: 0xa3a00154, 0.0.0.0/0.0.0.0:46030] timeout=300.0>) 2021-05-17 02:09:57,846 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x38a74c type=client open_count=1 channel=[id: 0x04180ad8, 0.0.0.0/0.0.0.0:52609] timeout=300.0>) 2021-05-17 02:09:59,878 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74d type=client open_count=1 channel=[id: 0x5de30d20, 0.0.0.0/0.0.0.0:55174] timeout=300.0>) 2021-05-17 02:10:24,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74e type=client open_count=1 channel=[id: 0xee7a1d07, 0.0.0.0/0.0.0.0:58226] timeout=300.0>) 2021-05-17 02:10:26,765 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x38a74f type=client open_count=1 channel=[id: 0x6750bbd0, 0.0.0.0/0.0.0.0:39185] timeout=300.0>) 2021-05-17 02:10:28,783 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a750 type=client open_count=1 channel=[id: 0x4614972f, 0.0.0.0/0.0.0.0:45999] timeout=300.0>) 2021-05-17 02:10:53,413 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a751 type=client open_count=1 channel=[id: 0xed6ab0d8, 0.0.0.0/0.0.0.0:59190] timeout=300.0>) 2021-05-17 02:10:55,476 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x38a752 type=client open_count=1 channel=[id: 0x88b36274, 0.0.0.0/0.0.0.0:42896] timeout=300.0>) 2021-05-17 02:10:57,509 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:11:17,526 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:11:31,512 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:11:31,536 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:11:31,573 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:11:45,163 | test | INFO | pool-9-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.68507808463 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 02:11:45,187 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:11:55,259 | test | INFO | pool-9-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:11:55,285 | test | INFO | pool-9-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.123000145 sec 2021-05-17 02:11:55,535 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:12:41,890 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:12:42,230 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 925929048 | 108210094 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 291696080 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1295415608 | 43904602 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 02:12:42,232 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:13:25,657 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:13:25,667 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:13:46,934 | test | INFO | pool-9-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 22781.131 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:13:47,032 | test | INFO | pool-9-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 157.415 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:13:47,088 | test | INFO | pool-9-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 149.829 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:13:47,357 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.5573822321 | 6437847040 | 4200837120 | 2583797760 / 7511588864 | 101251 / 245391 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.7689736532 | 6437847040 | 4204818432 | 2580123648 / 7511588864 | 100083 / 237747 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 98.7027851965 | 6437847040 | 4169306112 | 2603659264 / 7511588864 | 44688 / 115740 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.950154521 | 6437847040 | 3749453824 | 3241103360 / 9530753024 | 101065 / 266063 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.4823982216 | 6437847040 | 4187570176 | 2610003968 / 7511588864 | 47850 / 139215 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:14:02,081 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:15:28,723 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212902360 | 297653854 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 320312800 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 859893512 | 604754918 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:15:28,724 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #7 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:15:42,244 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 02:15:46,467 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 02:15:46,467 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before graceful_failover_recovery 2021-05-17 02:15:46,469 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:15:56,516 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:15:56,688 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 02:15:56,690 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:16:15,446 | test | INFO | pool-9-thread-7 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.......................................*.............................................................X | 16002.363 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:16:17,319 | test | INFO | pool-9-thread-23 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 3261.444 | | 129.262 | ..X | 129.262 | +---------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:16:17,549 | test | INFO | pool-9-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..*.................................................................................................X | 1730.482 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:17:13,114 | test | INFO | pool-9-thread-9 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 75.6370000839 seconds to complete 2021-05-17 02:17:13,121 | test | INFO | pool-9-thread-9 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:19:18,186 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0610001087189 seconds 2021-05-17 02:19:55,132 | test | INFO | pool-9-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 36.7649998665 seconds to complete 2021-05-17 02:19:55,141 | test | INFO | pool-9-thread-21 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:22:00,203 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0590000152588 seconds 2021-05-17 02:22:00,355 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:22:00,357 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:22:07,323 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:22:18,542 | test | INFO | pool-9-thread-5 | [table_view:display:72] Ops trend for bucket 'default' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 5248.537 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:19,367 | test | INFO | pool-9-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 4548.921 | +-------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:20,954 | test | INFO | pool-9-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..................................................................................*..................X | 5264.472 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:22:21,163 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,164 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,221 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:22:21,223 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:22:37,283 | test | INFO | pool-9-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.516701113408 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 79.2666666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 73.5736833361 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 75.8382774482 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.570019000633 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:22:37,308 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:42,368 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:47,421 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:53,615 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:22:58,811 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.95 2021-05-17 02:23:03,923 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 6.08 2021-05-17 02:23:09,203 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 10.22 2021-05-17 02:23:14,414 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 17.02 2021-05-17 02:23:19,528 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 22.04 2021-05-17 02:23:24,638 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 02:23:29,726 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.88 2021-05-17 02:23:36,828 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 35.21 2021-05-17 02:23:41,926 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 42.66 2021-05-17 02:23:47,078 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 46.59 2021-05-17 02:23:52,351 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 49.18 2021-05-17 02:23:57,914 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 53.33 2021-05-17 02:24:04,737 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 62.25 2021-05-17 02:24:10,038 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 72.08 2021-05-17 02:24:15,198 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-17 02:24:25,302 | test | INFO | pool-9-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:24:25,341 | test | INFO | pool-9-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 108.055999994 sec 2021-05-17 02:24:37,793 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:25:53,664 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 02:25:55,642 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 02:25:56,619 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:25:56,621 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:26:13,388 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 02:26:13,388 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 02:26:29,523 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:26:29,523 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:27:20,776 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1259999275 seconds to complete 2021-05-17 02:27:20,790 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1147.737s OK During the test, Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 7 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_7 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 8, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_graceful_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:27:30,144 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:27:30,836 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:27:31,256 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f24e type=client open_count=1 channel=[id: 0x19271d0a, 0.0.0.0/0.0.0.0:45923] timeout=300.0>) 2021-05-17 02:27:52,832 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f24f type=client open_count=1 channel=[id: 0xbf4f992a, 0.0.0.0/0.0.0.0:37739] timeout=300.0>) 2021-05-17 02:27:54,872 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x41f250 type=client open_count=1 channel=[id: 0x34142a6d, 0.0.0.0/0.0.0.0:33298] timeout=300.0>) 2021-05-17 02:27:56,918 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f251 type=client open_count=1 channel=[id: 0x92e46cbd, 0.0.0.0/0.0.0.0:52782] timeout=300.0>) 2021-05-17 02:28:23,749 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f252 type=client open_count=1 channel=[id: 0x89e7f94b, 0.0.0.0/0.0.0.0:41692] timeout=300.0>) 2021-05-17 02:28:25,782 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x41f253 type=client open_count=1 channel=[id: 0x97188be9, 0.0.0.0/0.0.0.0:55414] timeout=300.0>) 2021-05-17 02:28:27,813 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f254 type=client open_count=1 channel=[id: 0x75174782, 0.0.0.0/0.0.0.0:47245] timeout=300.0>) 2021-05-17 02:28:54,263 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f255 type=client open_count=1 channel=[id: 0x9cfe9fc7, 0.0.0.0/0.0.0.0:60196] timeout=300.0>) 2021-05-17 02:28:56,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x41f256 type=client open_count=1 channel=[id: 0x14dcd05f, 0.0.0.0/0.0.0.0:54912] timeout=300.0>) 2021-05-17 02:28:58,344 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f257 type=client open_count=1 channel=[id: 0x5e396793, 0.0.0.0/0.0.0.0:54428] timeout=300.0>) 2021-05-17 02:29:25,499 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f258 type=client open_count=1 channel=[id: 0xa32727fe, 0.0.0.0/0.0.0.0:49706] timeout=300.0>) 2021-05-17 02:29:27,516 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x41f259 type=client open_count=1 channel=[id: 0xe05d7c9a, 0.0.0.0/0.0.0.0:43710] timeout=300.0>) 2021-05-17 02:29:29,562 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25a type=client open_count=1 channel=[id: 0x407b8769, 0.0.0.0/0.0.0.0:51359] timeout=300.0>) 2021-05-17 02:29:55,552 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25b type=client open_count=1 channel=[id: 0xdb06c792, 0.0.0.0/0.0.0.0:48522] timeout=300.0>) 2021-05-17 02:29:57,581 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x41f25c type=client open_count=1 channel=[id: 0x6fd5d9a1, 0.0.0.0/0.0.0.0:40022] timeout=300.0>) 2021-05-17 02:29:59,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:30:19,052 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:30:34,578 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:30:34,601 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:30:34,638 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:30:48,232 | test | INFO | pool-10-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.14020701035 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 02:30:48,272 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:30:58,404 | test | INFO | pool-10-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:30:58,457 | test | INFO | pool-10-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2239999771 sec 2021-05-17 02:30:59,020 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:31:54,210 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:31:54,990 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 805387936 | 153067684 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 283598864 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 720892248 | 122986010 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 02:31:54,990 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:32:50,368 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:32:50,377 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:33:15,266 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 165.054 | | 74.319 | ............................................*X | 74.319 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:33:16,631 | test | INFO | pool-10-thread-12 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **..................................***.................................................................**X | 21515.831 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:33:17,434 | test | INFO | pool-10-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................**X | 156.557 | | 62.324 | ......................................**....X | 67.636 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:33:17,734 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.2944621938 | 6437847040 | 4182282240 | 2588782592 / 7511588864 | 59333 / 179322 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.4284574998 | 6437847040 | 4161486848 | 2610003968 / 7511588864 | 58727 / 157168 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.2744458497 | 6437847040 | 4178894848 | 2587127808 / 7511588864 | 100400 / 303800 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.4800086665 | 6437847040 | 3746578432 | 3203096576 / 9530753024 | 58960 / 146429 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 98.3586301324 | 6437847040 | 4191698944 | 2611060736 / 7511588864 | 101600 / 301000 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:33:36,380 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:35:27,805 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211093320 | 311551038 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319196752 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 811598376 | 644304730 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:35:27,808 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #8 test_data_load_collections_with_graceful_failover_recovery ========= 2021-05-17 02:35:43,335 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_recovery 2021-05-17 02:35:43,336 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_recovery 2021-05-17 02:35:43,338 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:679] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:36:42,918 | test | INFO | pool-10-thread-25 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 59.2319998741 seconds to complete 2021-05-17 02:36:42,924 | test | INFO | pool-10-thread-25 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:38:48,036 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.108999967575 seconds 2021-05-17 02:39:31,638 | test | INFO | pool-10-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 43.3489999771 seconds to complete 2021-05-17 02:39:31,645 | test | INFO | pool-10-thread-24 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 02:41:36,704 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 02:41:36,704 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 02:41:36,706 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:41:43,438 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:42:00,408 | test | INFO | pool-10-thread-5 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 3090.114 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:01,220 | test | INFO | pool-10-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................X | 23522.986 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:42:02,404 | test | INFO | pool-10-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................*X | 3953.639 | +-------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:02,653 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,653 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,710 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 02:42:02,711 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 02:42:19,941 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:42:20,191 | test | INFO | pool-10-thread-12 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.549972501375 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 74.4649951591 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 64.9466666667 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 80.2346705778 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.418326361227 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 02:42:20,260 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:25,348 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:26,877 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:42:31,230 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:36,454 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:41,559 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:41,572 | test | INFO | pool-10-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 4142.002 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:42,198 | test | INFO | pool-10-thread-2 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 13150.754 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:42:44,584 | test | INFO | pool-10-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 5335.378 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:42:46,663 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:51,946 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:42:57,023 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:43:02,095 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.46 2021-05-17 02:43:07,263 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 1.71 2021-05-17 02:43:12,523 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 6.32 2021-05-17 02:43:17,704 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 11.39 2021-05-17 02:43:22,829 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 15.79 2021-05-17 02:43:27,970 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 19.43 2021-05-17 02:43:33,131 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 24.17 2021-05-17 02:43:39,885 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 02:43:45,141 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 30.76 2021-05-17 02:43:50,326 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 36.88 2021-05-17 02:43:55,489 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 40.28 2021-05-17 02:44:00,766 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 43.74 2021-05-17 02:44:06,019 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 47.61 2021-05-17 02:44:11,233 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 51.48 2021-05-17 02:44:17,007 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 55.35 2021-05-17 02:44:22,217 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 61.94 2021-05-17 02:44:27,443 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 67.77 2021-05-17 02:44:32,601 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: running, progress: 73.72 2021-05-17 02:44:42,782 | test | INFO | pool-10-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:44:42,819 | test | INFO | pool-10-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 142.628999949 sec 2021-05-17 02:44:55,292 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:46:12,086 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 02:46:12,086 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 02:46:27,497 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 98% 2021-05-17 02:46:27,499 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 95% 2021-05-17 02:46:38,974 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 02:46:38,974 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 02:47:46,826 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.125 seconds to complete 2021-05-17 02:47:46,841 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok During the test, ---------------------------------------------------------------------- Ran 1 test in 1226.031s Remote Connections: 202, Disconnections: 202 SDK Connections: 30, Disconnections: 30 OK summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 8 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_8 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 9, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 02:47:56,348 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:47:57,046 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:47:57,463 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3215 type=client open_count=1 channel=[id: 0x91e65596, 0.0.0.0/0.0.0.0:55411] timeout=300.0>) 2021-05-17 02:48:18,216 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3216 type=client open_count=1 channel=[id: 0x3c8d7ce6, 0.0.0.0/0.0.0.0:53054] timeout=300.0>) 2021-05-17 02:48:20,233 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x4b3217 type=client open_count=1 channel=[id: 0x310b57cc, 0.0.0.0/0.0.0.0:58618] timeout=300.0>) 2021-05-17 02:48:22,252 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b3218 type=client open_count=1 channel=[id: 0x0dc7c7bc, 0.0.0.0/0.0.0.0:41750] timeout=300.0>) 2021-05-17 02:48:46,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b3219 type=client open_count=1 channel=[id: 0x445c061e, 0.0.0.0/0.0.0.0:57257] timeout=300.0>) 2021-05-17 02:48:48,572 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x4b321a type=client open_count=1 channel=[id: 0xef98fa87, 0.0.0.0/0.0.0.0:34609] timeout=300.0>) 2021-05-17 02:48:50,591 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321b type=client open_count=1 channel=[id: 0xc376d9db, 0.0.0.0/0.0.0.0:56538] timeout=300.0>) 2021-05-17 02:49:14,726 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321c type=client open_count=1 channel=[id: 0xbc7a599e, 0.0.0.0/0.0.0.0:51960] timeout=300.0>) 2021-05-17 02:49:16,743 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x4b321d type=client open_count=1 channel=[id: 0x69dcde71, 0.0.0.0/0.0.0.0:44541] timeout=300.0>) 2021-05-17 02:49:18,763 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b321e type=client open_count=1 channel=[id: 0xaf60496a, 0.0.0.0/0.0.0.0:33524] timeout=300.0>) 2021-05-17 02:49:42,884 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b321f type=client open_count=1 channel=[id: 0x8e48e806, 0.0.0.0/0.0.0.0:58203] timeout=300.0>) 2021-05-17 02:49:44,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x4b3220 type=client open_count=1 channel=[id: 0x8d2b7f95, 0.0.0.0/0.0.0.0:54513] timeout=300.0>) 2021-05-17 02:49:46,920 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3221 type=client open_count=1 channel=[id: 0xfd6f2423, 0.0.0.0/0.0.0.0:49449] timeout=300.0>) 2021-05-17 02:50:11,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3222 type=client open_count=1 channel=[id: 0x25198c38, 0.0.0.0/0.0.0.0:43300] timeout=300.0>) 2021-05-17 02:50:13,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x4b3223 type=client open_count=1 channel=[id: 0xd9046e3b, 0.0.0.0/0.0.0.0:52173] timeout=300.0>) 2021-05-17 02:50:15,091 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 02:50:32,338 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 02:50:46,025 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 02:50:46,046 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:50:46,084 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:51:00,042 | test | INFO | pool-11-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 15.155 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------+--------------+ 2021-05-17 02:51:00,069 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 02:51:10,142 | test | INFO | pool-11-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 02:51:10,168 | test | INFO | pool-11-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1259999275 sec 2021-05-17 02:51:10,418 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 02:52:07,040 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 02:52:07,415 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 737052152 | 153291653 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 304136576 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 806693864 | 118008106 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 02:52:07,417 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 02:52:47,184 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:52:47,197 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:53:08,397 | test | INFO | pool-11-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **.............................................................................................*.......*X | 240.802 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:53:08,477 | test | INFO | pool-11-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **........................................................................*...........................*X | 221.299 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 02:53:08,594 | test | INFO | pool-11-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.....................................*...............................................................*X | 17316.022 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:53:08,887 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.7642613522 | 6437847040 | 4221870080 | 2556145664 / 7511588864 | 36218 / 107423 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.6594721819 | 6437847040 | 4207136768 | 2555310080 / 7511588864 | 30526 / 80328 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9216679722 | 6437847040 | 4211765248 | 2554540032 / 7511588864 | 39364 / 105093 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.7661032132 | 6437847040 | 3775979520 | 3186417664 / 9530753024 | 35892 / 87598 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.5043204501 | 6437847040 | 4240412672 | 2558140416 / 7511588864 | 35722 / 89544 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 02:53:23,844 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 02:54:50,667 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 210629208 | 314634507 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319167856 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 846689368 | 630570239 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 02:54:50,667 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #9 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 02:55:00,891 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 02:55:02,509 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 02:55:02,509 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_recovery 2021-05-17 02:55:02,510 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 02:55:10,747 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 02:55:10,920 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 02:55:10,921 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 02:55:22,260 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket2:idQU6seGSZGDX4bL-7-358000:KZm6BjaG6B12S-7-622000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3290 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3271 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3227 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3349 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3209 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3309 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3285 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3262 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3240 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3267 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3300 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3245 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3222 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3200 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3345 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3283 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3216 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3259 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3338 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3350 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3331 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3231 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3253 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3234 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3256 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3299 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3356 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3378 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3213 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:22,263 | test | INFO | pool-11-thread-14 | [table_view:display:72] Keys failed in bucket2:idQU6seGSZGDX4bL-7-358000:KZm6BjaG6B12S-7-622000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3290 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3271 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3227 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3205 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3349 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3209 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3309 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3285 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3262 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3240 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3267 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3245 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3300 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3222 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3200 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3345 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3283 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3216 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3259 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3338 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3274 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-3350 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3331 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3231 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3253 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3234 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3256 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3299 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3213 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3378 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-3356 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:22,733 | test | INFO | pool-11-thread-11 | [table_view:display:72] Keys failed in default:T4-10-507000:aPIoDiKGcZg25-10-507000 +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-308 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-348 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-204 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-325 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-226 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-366 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-201 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-223 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-344 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-301 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-323 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-244 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-266 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-387 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-263 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-241 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-382 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-360 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-339 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-217 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-239 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-379 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-336 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-212 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-257 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-235 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-375 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-298 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-230 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-252 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-373 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-330 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-275 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-394 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-270 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-391 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-291 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | +----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:23,832 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in default:n%Zit8F6-9-794000:r6N%AEM8qJhkd-9-994000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4559 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4515 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4438 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4537 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4437 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4415 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4459 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4519 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4596 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4574 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4450 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4593 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4472 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4572 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4550 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4432 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4531 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4410 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4477 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4455 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4406 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4403 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4568 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4524 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4528 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4485 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4464 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4585 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4461 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4561 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4489 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4446 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4468 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4424 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4443 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4421 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:23,835 | test | INFO | pool-11-thread-13 | [table_view:display:72] Keys failed in default:n%Zit8F6-9-794000:r6N%AEM8qJhkd-9-994000 +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4580 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4438 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4537 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4515 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4559 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4459 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4415 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4437 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4519 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4574 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4596 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4593 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4472 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4450 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4572 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4550 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4410 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4432 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4531 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4455 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4477 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4406 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4403 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4524 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4568 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4528 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4485 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4585 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4464 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4461 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4561 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4489 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4424 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4468 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4446 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4443 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-4421 | com.couchbase.client.core.error.RequestCanceledException | InsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +-----------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:24,118 | test | INFO | pool-11-thread-24 | [table_view:display:72] Keys failed in default:scope1:TjvJ8R-10-499000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 02:55:24,273 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 37 docs from 0 to 5000 2021-05-17 02:55:25,299 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 31 docs from 0 to 5000 2021-05-17 02:55:25,321 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 37 docs from 0 to 500 2021-05-17 02:55:25,325 | test | ERROR | pool-11-thread-30 | [task:call:3809] Failed to load 41 docs from 500 to 980 2021-05-17 02:55:30,200 | test | INFO | pool-11-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *......*..............................................................................................X | 4225.989 | | 1553.709 | ...................................X | 1553.709 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:55:31,019 | test | INFO | pool-11-thread-26 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........................................*..........................................................X | 4299.946 | | 35.806 | *X | 35.806 | +--------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 02:55:31,961 | test | INFO | pool-11-thread-15 | [table_view:display:72] Ops trend for bucket 'default' +----------+--------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....................................*...............................................................X | 12208.670 | | 9994.626 | ................................................................................*X | 9994.626 | +----------+--------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 02:55:41,756 | test | INFO | pool-11-thread-21 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.21099996567 seconds to complete 2021-05-17 02:55:41,762 | test | INFO | pool-11-thread-21 | [common_lib:sleep:22] Sleep 8.21099996567 seconds. Reason: Wait after rebalance complete 2021-05-17 02:57:50,030 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-05-17 02:58:01,983 | test | INFO | pool-11-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.31200003624 seconds to complete 2021-05-17 02:58:01,989 | test | INFO | pool-11-thread-24 | [common_lib:sleep:22] Sleep 8.31200003624 seconds. Reason: Wait after rebalance complete 2021-05-17 03:00:10,361 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 03:00:10,361 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:00:10,362 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:00:17,944 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:00:33,278 | test | INFO | pool-11-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........................*...........................................................*...............X | 5895.362 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:00:34,128 | test | INFO | pool-11-thread-4 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*..........................................................................................X | 7676.942 | | 4187.511 | .....................................................X | 4187.511 | +----------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:00:35,598 | test | INFO | pool-11-thread-6 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *..........*...............................................................................**...........X | 22997.150 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:00:35,776 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,778 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,917 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:00:35,918 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:00:52,456 | test | INFO | pool-11-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.156661444619 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 62.8633333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 41.7180572686 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 58.4604066346 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.313333333333 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:00:52,519 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:00:57,617 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:02,677 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:07,736 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:01:13,023 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.41 2021-05-17 03:01:18,391 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 11.08 2021-05-17 03:01:23,575 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.78 2021-05-17 03:01:28,713 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 23.08 2021-05-17 03:01:33,834 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 26.74 2021-05-17 03:01:38,980 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 03:01:45,661 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 39.09 2021-05-17 03:01:50,799 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 49.26 2021-05-17 03:01:55,936 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 55.71 2021-05-17 03:02:01,039 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 59.86 2021-05-17 03:02:06,121 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.57 2021-05-17 03:02:11,207 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-17 03:02:16,780 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 72.63 2021-05-17 03:02:21,926 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 79.01 2021-05-17 03:02:27,055 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 82.77 2021-05-17 03:02:32,196 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 86.22 2021-05-17 03:02:37,309 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 88.31 2021-05-17 03:02:42,405 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 92.5 2021-05-17 03:02:48,354 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: running, progress: 95.32 2021-05-17 03:02:58,470 | test | INFO | pool-11-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:02:58,517 | test | INFO | pool-11-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 126.061999798 sec 2021-05-17 03:02:58,519 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 03:03:01,293 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 03:03:02,628 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:03:02,628 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:03:20,252 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:03:20,253 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:03:39,713 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:03:39,714 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:04:38,119 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1569998264 seconds to complete 2021-05-17 03:04:38,134 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1011.322s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 9 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_9 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=full,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'full', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 10, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:04:48,246 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:04:49,030 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:04:49,575 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548842 type=client open_count=1 channel=[id: 0x8488680e, 0.0.0.0/0.0.0.0:46827] timeout=300.0>) 2021-05-17 03:05:10,687 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548843 type=client open_count=1 channel=[id: 0xc9c85073, 0.0.0.0/0.0.0.0:52026] timeout=300.0>) 2021-05-17 03:05:12,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x548844 type=client open_count=1 channel=[id: 0xdf9a814e, 0.0.0.0/0.0.0.0:43598] timeout=300.0>) 2021-05-17 03:05:14,750 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548845 type=client open_count=1 channel=[id: 0x1e9b8405, 0.0.0.0/0.0.0.0:38336] timeout=300.0>) 2021-05-17 03:05:40,053 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548846 type=client open_count=1 channel=[id: 0xcedd59ed, 0.0.0.0/0.0.0.0:55751] timeout=300.0>) 2021-05-17 03:05:42,104 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x548847 type=client open_count=1 channel=[id: 0xd08a7ba5, 0.0.0.0/0.0.0.0:37135] timeout=300.0>) 2021-05-17 03:05:44,121 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x548848 type=client open_count=1 channel=[id: 0xf999201b, 0.0.0.0/0.0.0.0:50063] timeout=300.0>) 2021-05-17 03:06:08,315 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x548849 type=client open_count=1 channel=[id: 0x2ca79a91, 0.0.0.0/0.0.0.0:34538] timeout=300.0>) 2021-05-17 03:06:10,335 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x54884a type=client open_count=1 channel=[id: 0x4cb73f93, 0.0.0.0/0.0.0.0:42994] timeout=300.0>) 2021-05-17 03:06:12,352 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884b type=client open_count=1 channel=[id: 0x302ccf8f, 0.0.0.0/0.0.0.0:53506] timeout=300.0>) 2021-05-17 03:06:36,582 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884c type=client open_count=1 channel=[id: 0x73a70209, 0.0.0.0/0.0.0.0:38020] timeout=300.0>) 2021-05-17 03:06:38,601 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x54884d type=client open_count=1 channel=[id: 0x4b56b36b, 0.0.0.0/0.0.0.0:39241] timeout=300.0>) 2021-05-17 03:06:40,618 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x54884e type=client open_count=1 channel=[id: 0xf88177e3, 0.0.0.0/0.0.0.0:51214] timeout=300.0>) 2021-05-17 03:07:04,576 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x54884f type=client open_count=1 channel=[id: 0x4ccabc3d, 0.0.0.0/0.0.0.0:52710] timeout=300.0>) 2021-05-17 03:07:06,595 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x548850 type=client open_count=1 channel=[id: 0xb998f3bd, 0.0.0.0/0.0.0.0:48271] timeout=300.0>) 2021-05-17 03:07:08,611 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:07:25,951 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:07:39,640 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:07:39,661 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:07:39,700 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:07:52,983 | test | INFO | pool-12-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+-----+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+-----+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+-----+--------------+ 2021-05-17 03:07:53,009 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:08:03,082 | test | INFO | pool-12-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:08:03,108 | test | INFO | pool-12-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1240000725 sec 2021-05-17 03:08:03,384 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:08:51,065 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:08:51,414 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 960411712 | 102400167 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 301818688 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1227237224 | 54793741 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 03:08:51,415 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:09:33,733 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:09:33,746 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:09:53,782 | test | INFO | pool-12-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *....**................................................................................................X | 27175.967 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:09:54,358 | test | INFO | pool-12-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................X | 148.445 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:09:54,756 | test | INFO | pool-12-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *..............................................**......................................................X | 132.650 | +-------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:09:55,026 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 71.1891765959 | 6437847040 | 4303298560 | 2456530944 / 7511588864 | 58519 / 136186 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 70.1332623488 | 6437847040 | 4235509760 | 2530041856 / 7511588864 | 57371 / 146457 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 66.7188072021 | 6437847040 | 4247871488 | 2515165184 / 7511588864 | 56769 / 139257 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 68.4213133431 | 6437847040 | 3825758208 | 3108995072 / 9530753024 | 57804 / 169097 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7896739947 | 6437847040 | 4142186496 | 2637008896 / 7511588864 | 56976 / 169147 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:10:10,280 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:11:37,010 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212554840 | 303361380 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319351376 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 824483112 | 608284650 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:11:37,012 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #10 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:11:47,178 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 03:11:47,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:11:47,180 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:11:59,056 | test | INFO | pool-12-thread-3 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.22000002861 seconds to complete 2021-05-17 03:11:59,062 | test | INFO | pool-12-thread-3 | [common_lib:sleep:22] Sleep 8.22000002861 seconds. Reason: Wait after rebalance complete 2021-05-17 03:14:07,345 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 03:14:19,035 | test | INFO | pool-12-thread-20 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.125 seconds to complete 2021-05-17 03:14:19,039 | test | INFO | pool-12-thread-20 | [common_lib:sleep:22] Sleep 8.125 seconds. Reason: Wait after rebalance complete 2021-05-17 03:16:27,224 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0550000667572 seconds 2021-05-17 03:16:27,224 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:16:27,226 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:16:35,575 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:16:54,493 | test | INFO | pool-12-thread-25 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 19648.509 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:16:56,085 | test | INFO | pool-12-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 2885.009 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:16:56,171 | test | INFO | pool-12-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 2170.320 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:16:56,273 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,275 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,332 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:16:56,332 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:17:08,144 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:17:08,309 | test | INFO | pool-12-thread-2 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 1.09331511141 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 89.5075269177 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 75.545 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 85.1386822263 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.393320222659 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:17:08,368 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:13,450 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:16,107 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:17:18,596 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:23,694 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:28,828 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:35,213 | test | INFO | pool-12-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***...................................................................................................*X | 14316.942 | +-------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:17:35,779 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:17:37,585 | test | INFO | pool-12-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 3152.126 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:17:37,789 | test | INFO | pool-12-thread-19 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................**X | 4878.498 | +-------+------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:17:40,937 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 4.15 2021-05-17 03:17:46,062 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 6.92 2021-05-17 03:17:51,164 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 9.71 2021-05-17 03:17:56,269 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 12.93 2021-05-17 03:18:01,417 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 16.19 2021-05-17 03:18:06,565 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 20.09 2021-05-17 03:18:11,743 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 24.34 2021-05-17 03:18:16,851 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 27.79 2021-05-17 03:18:21,921 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 30.59 2021-05-17 03:18:27,023 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 03:18:32,119 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 36.96 2021-05-17 03:18:37,272 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 44.98 2021-05-17 03:18:42,404 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 51.18 2021-05-17 03:18:47,496 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 57.17 2021-05-17 03:18:52,598 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 62.18 2021-05-17 03:18:57,776 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 65.8 2021-05-17 03:19:04,361 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 67.74 2021-05-17 03:19:09,483 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 74.6 2021-05-17 03:19:14,622 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 78.86 2021-05-17 03:19:19,742 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 82.11 2021-05-17 03:19:24,851 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 85.86 2021-05-17 03:19:29,957 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 89.35 2021-05-17 03:19:35,088 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 92.69 2021-05-17 03:19:40,167 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 95.36 2021-05-17 03:19:45,262 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 97.76 2021-05-17 03:19:51,401 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: running, progress: 100.0 2021-05-17 03:20:01,496 | test | INFO | pool-12-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:20:01,533 | test | INFO | pool-12-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 173.223999977 sec 2021-05-17 03:20:02,522 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:20:02,523 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:20:16,153 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:20:16,153 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:20:36,926 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:20:36,926 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:21:45,138 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1400001049 seconds to complete 2021-05-17 03:21:45,153 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1027.066s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 10 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_10 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=before,scrape_interval=5,rebalance_moves_per_node=32,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'before', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11', 'nodes_init': '5', 'rebalance_moves_per_node': '32', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 11, 'scrape_interval': '5', 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:21:57,653 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:21:59,525 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:22:01,632 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6024 type=client open_count=1 channel=[id: 0x6e664d4d, 0.0.0.0/0.0.0.0:48131] timeout=300.0>) 2021-05-17 03:22:25,556 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6025 type=client open_count=1 channel=[id: 0xbb56cd7e, 0.0.0.0/0.0.0.0:60859] timeout=300.0>) 2021-05-17 03:22:27,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x5e6026 type=client open_count=1 channel=[id: 0xc18f2465, 0.0.0.0/0.0.0.0:37907] timeout=300.0>) 2021-05-17 03:22:29,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6027 type=client open_count=1 channel=[id: 0x8b7f8d9f, 0.0.0.0/0.0.0.0:48404] timeout=300.0>) 2021-05-17 03:22:54,755 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6028 type=client open_count=1 channel=[id: 0x290b65f5, 0.0.0.0/0.0.0.0:59097] timeout=300.0>) 2021-05-17 03:22:56,798 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x5e6029 type=client open_count=1 channel=[id: 0xab9ac5a6, 0.0.0.0/0.0.0.0:47605] timeout=300.0>) 2021-05-17 03:22:58,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602a type=client open_count=1 channel=[id: 0xa9543501, 0.0.0.0/0.0.0.0:41979] timeout=300.0>) 2021-05-17 03:23:25,428 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602b type=client open_count=1 channel=[id: 0x0f316aa7, 0.0.0.0/0.0.0.0:40947] timeout=300.0>) 2021-05-17 03:23:27,456 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x5e602c type=client open_count=1 channel=[id: 0x2c6f9bdd, 0.0.0.0/0.0.0.0:59938] timeout=300.0>) 2021-05-17 03:23:29,493 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602d type=client open_count=1 channel=[id: 0xa69678e5, 0.0.0.0/0.0.0.0:50287] timeout=300.0>) 2021-05-17 03:23:55,180 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602e type=client open_count=1 channel=[id: 0x395730ed, 0.0.0.0/0.0.0.0:46932] timeout=300.0>) 2021-05-17 03:23:57,219 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x5e602f type=client open_count=1 channel=[id: 0xe624228d, 0.0.0.0/0.0.0.0:35234] timeout=300.0>) 2021-05-17 03:23:59,265 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6030 type=client open_count=1 channel=[id: 0x0a24ccfc, 0.0.0.0/0.0.0.0:44145] timeout=300.0>) 2021-05-17 03:24:26,585 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6031 type=client open_count=1 channel=[id: 0x7f9f1660, 0.0.0.0/0.0.0.0:46700] timeout=300.0>) 2021-05-17 03:24:28,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x5e6032 type=client open_count=1 channel=[id: 0x83d8b775, 0.0.0.0/0.0.0.0:43378] timeout=300.0>) 2021-05-17 03:24:30,667 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:24:51,571 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:25:08,979 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:25:09,055 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:25:09,197 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:25:23,779 | test | INFO | pool-13-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.23005383423 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:25:23,849 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:25:34,015 | test | INFO | pool-13-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:25:34,051 | test | INFO | pool-13-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2709999084 sec 2021-05-17 03:25:34,437 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:26:29,555 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:26:29,898 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 685582936 | 161103560 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 302167600 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 836786712 | 92346589 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 03:26:29,900 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:27:20,605 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:27:20,614 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:27:42,378 | test | INFO | pool-13-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **....................................................................................................*X | 140.792 | | 114.856 | ................................................................................X | 114.856 | +---------+----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:27:42,549 | test | INFO | pool-13-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | **..................................................................................................**..X | 122.468 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:27:42,819 | test | INFO | pool-13-thread-13 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................**................................................................X | 25623.271 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:27:43,108 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.9732078568 | 6437847040 | 4172472320 | 2597888000 / 7511588864 | 71649 / 214121 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.7896897503 | 6437847040 | 4108689408 | 2670399488 / 7511588864 | 69525 / 184675 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.9461596702 | 6437847040 | 4104855552 | 2659078144 / 7511588864 | 71754 / 219935 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 97.4006955885 | 6437847040 | 3806674944 | 3139280896 / 9530753024 | 71581 / 145178 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.9733754888 | 6437847040 | 4141879296 | 2644598784 / 7511588864 | 70887 / 175996 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:27:58,795 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:29:26,259 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 211874840 | 319990167 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319636832 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 836021960 | 619601592 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:29:26,259 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #11 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:29:36,523 | test | INFO | MainThread | [collections_rebalance:setUp:58] Changing scrape interval to 5 2021-05-17 03:29:38,176 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 32} 2021-05-17 03:29:38,177 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load before hard_failover_recovery 2021-05-17 03:29:38,177 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:29:47,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:29:47,546 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:29:47,548 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:29:58,151 | test | INFO | pool-13-thread-28 | [table_view:display:72] Keys failed in bucket2:q7GNTf-34-521000:tM17GyXfKDR2-34-523000 +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | +--------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:00,816 | test | INFO | pool-13-thread-20 | [table_view:display:72] Keys failed in default:scope1:Wzooi-34-545000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-436 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-412 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-497 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,459 | test | INFO | pool-13-thread-15 | [table_view:display:72] Keys failed in default:_default:4XNodbPcjHD-34-543000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE]', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,480 | test | INFO | pool-13-thread-7 | [table_view:display:72] Keys failed in default:fALratH-34-553000:E3a5YEUM45lDcfbakQMO-34-555000 +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-38 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-16 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-10 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-55 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-34 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-78 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-109 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-106 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-95 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-53 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-146 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-124 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-168 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-103 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-189 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-164 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-121 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-143 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-185 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-183 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-161 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-49 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-0 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-89 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-83 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-40 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-137 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-159 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-115 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-86 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-90 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-177 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-155 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-71 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-110 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None', 'cas': 0, 'value': {}} | | test_collections-132 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-196 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-9 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-150 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-172 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | | test_collections-5 | {'error': 'com.couchbase.client.core.error.RequestCanceledException | RemoveRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS]', 'cas': 0, 'value': {}} | +----------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,849 | test | INFO | pool-13-thread-27 | [table_view:display:72] Keys failed in default:scope1:Wzooi-34-545000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-917 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-926 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-904 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-966 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-901 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-930 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-970 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:01,865 | test | INFO | pool-13-thread-23 | [table_view:display:72] Keys failed in default:fALratH-34-553000:LQ-34-553000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-667 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-699 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-674 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-690 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:02,388 | test | INFO | pool-13-thread-11 | [table_view:display:72] Keys failed in default:scope1:OrDK1mLiJa8Ns5-34-546000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-709 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-808 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-847 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-826 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-749 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-848 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-804 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-724 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-802 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-887 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-700 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-722 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-767 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-745 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-888 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-841 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-863 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-786 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-783 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-882 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-761 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-817 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-839 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-737 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-738 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-879 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-854 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-799 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-778 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-811 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-852 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-731 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-795 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-894 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-774 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-870 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-891 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:2 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-790 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:03,088 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 1 docs from 0 to 3 2021-05-17 03:30:03,092 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 19 docs from 0 to 500 2021-05-17 03:30:03,094 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 41 docs from 0 to 500 2021-05-17 03:30:03,094 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 38 docs from 500 to 980 2021-05-17 03:30:03,104 | test | INFO | pool-13-thread-2 | [table_view:display:72] Keys failed in default:fALratH-34-553000:T5hcTCsaKMV%X-34-554000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-609 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-508 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-627 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-605 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-507 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-667 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-502 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-645 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-668 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-547 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-569 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-525 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-588 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-600 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-622 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-520 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-542 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-565 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-584 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-640 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-662 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-560 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-582 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-638 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-616 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-656 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-634 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-558 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-536 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-514 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-613 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-511 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-533 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-699 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-674 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-597 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-576 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-631 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-554 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-653 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:1 | retryReasons:[ENDPOINT_NOT_WRITABLE] | | test_collections-573 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-551 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-671 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-690 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | | test_collections-591 | com.couchbase.client.core.error.RequestCanceledException | UpsertRequest, Reason: NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | 172.23.107.131:11210 | reason:NO_MORE_RETRIES (CHANNEL_CLOSED_WHILE_IN_FLIGHT) | retryAttempts:0 | retryReasons:None | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 03:30:03,381 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 45 docs from 500 to 980 2021-05-17 03:30:03,382 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 7 docs from 500 to 980 2021-05-17 03:30:03,384 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 3 docs from 0 to 500 2021-05-17 03:30:03,384 | test | ERROR | pool-13-thread-29 | [task:call:3809] Failed to load 45 docs from 500 to 980 2021-05-17 03:30:12,525 | test | INFO | pool-13-thread-21 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.**...................................................................................................X | 5789.645 | | 4747.162 | ................................................................................X | 4747.162 | | 1236.179 | ....................*X | 1236.179 | | 61.817 | X | 61.817 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:30:12,890 | test | INFO | pool-13-thread-1 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***....................................................................................................X | 4058.789 | | 163.608 | ...**....X | 356.264 | +---------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:30:12,933 | test | INFO | pool-13-thread-4 | [table_view:display:72] Ops trend for bucket 'default' +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***..............................................................................*.....**.................X | 19342.030 | +-------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:30:15,664 | test | INFO | pool-13-thread-5 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.25499987602 seconds to complete 2021-05-17 03:30:15,670 | test | INFO | pool-13-thread-5 | [common_lib:sleep:22] Sleep 8.25499987602 seconds. Reason: Wait after rebalance complete 2021-05-17 03:32:23,992 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0640001296997 seconds 2021-05-17 03:32:36,191 | test | INFO | pool-13-thread-24 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.48099994659 seconds to complete 2021-05-17 03:32:36,197 | test | INFO | pool-13-thread-24 | [common_lib:sleep:22] Sleep 8.48099994659 seconds. Reason: Wait after rebalance complete 2021-05-17 03:34:44,734 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.0539999008179 seconds 2021-05-17 03:34:44,736 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:34:44,736 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:34:50,586 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:35:05,710 | test | INFO | pool-13-thread-7 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+--------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+--------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *....................................................................................................X | 4248.947 | | 2778.516 | ................................................................X | 2778.516 | +----------+--------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:35:07,174 | test | INFO | pool-13-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **..........................................................................*..........................X | 6734.454 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:35:07,898 | test | INFO | pool-13-thread-8 | [table_view:display:72] Ops trend for bucket 'default' +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.....................................*.................................*..............................X | 22955.942 | | 22928.688 | ..................................................................................................X | 22928.688 | +-----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:35:08,000 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,000 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,055 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:35:08,055 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:35:23,428 | test | INFO | pool-13-thread-10 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 0.516683889463 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 67.7646980135 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 67.9155971866 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 78.3710588786 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 2.86990433652 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:35:23,453 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:28,510 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:33,569 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:38,628 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:43,711 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:35:48,907 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.36 2021-05-17 03:35:54,055 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 7.21 2021-05-17 03:35:59,186 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 13.65 2021-05-17 03:36:04,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 17.92 2021-05-17 03:36:09,650 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 20.3 2021-05-17 03:36:14,720 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.58 2021-05-17 03:36:19,782 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 03:36:24,852 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.4 2021-05-17 03:36:29,924 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.04 2021-05-17 03:36:35,003 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.77 2021-05-17 03:36:40,075 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 47.24 2021-05-17 03:36:45,187 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 49.38 2021-05-17 03:36:50,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 53.33 2021-05-17 03:36:55,415 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 58.49 2021-05-17 03:37:02,173 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 66.9 2021-05-17 03:37:07,302 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 73.14 2021-05-17 03:37:12,372 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 77.17 2021-05-17 03:37:18,305 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-17 03:37:28,405 | test | INFO | pool-13-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:37:28,443 | test | INFO | pool-13-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 125.015000105 sec 2021-05-17 03:37:28,444 | test | INFO | MainThread | [collections_rebalance:tearDown:84] Reverting prometheus settings back to default 2021-05-17 03:37:30,443 | test | INFO | MainThread | [cluster_ready_functions:set_rebalance_moves_per_nodes:129] Changed Rebalance settings: {u'rebalanceMovesPerNode': 4} 2021-05-17 03:37:31,430 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:37:31,431 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:37:44,351 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:37:44,352 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:38:04,990 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:38:04,990 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:39:03,371 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.128000021 seconds to complete 2021-05-17 03:39:03,387 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 1038.075s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 11 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_11 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_recovery,nodes_init=5,nodes_failover=2,recovery_type=delta,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=True,GROUP=failover_with_collection_crud' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '2', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'recovery_type': 'delta', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'True', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 12, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_data_load_collections_with_hard_failover_recovery (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:39:12,762 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:39:13,441 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:39:13,861 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8a type=client open_count=1 channel=[id: 0x472be428, 0.0.0.0/0.0.0.0:49472] timeout=300.0>) 2021-05-17 03:39:34,674 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8b type=client open_count=1 channel=[id: 0x803d9297, 0.0.0.0/0.0.0.0:52063] timeout=300.0>) 2021-05-17 03:39:36,693 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x67ff8c type=client open_count=1 channel=[id: 0x4a0d19be, 0.0.0.0/0.0.0.0:48917] timeout=300.0>) 2021-05-17 03:39:38,713 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8d type=client open_count=1 channel=[id: 0xbeb90def, 0.0.0.0/0.0.0.0:52125] timeout=300.0>) 2021-05-17 03:40:03,105 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8e type=client open_count=1 channel=[id: 0x302f9760, 0.0.0.0/0.0.0.0:43281] timeout=300.0>) 2021-05-17 03:40:05,125 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x67ff8f type=client open_count=1 channel=[id: 0xd3dfedfe, 0.0.0.0/0.0.0.0:34933] timeout=300.0>) 2021-05-17 03:40:07,142 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff90 type=client open_count=1 channel=[id: 0x264ae60c, 0.0.0.0/0.0.0.0:43863] timeout=300.0>) 2021-05-17 03:40:31,665 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff91 type=client open_count=1 channel=[id: 0x278def2f, 0.0.0.0/0.0.0.0:52322] timeout=300.0>) 2021-05-17 03:40:33,691 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x67ff92 type=client open_count=1 channel=[id: 0x322c925f, 0.0.0.0/0.0.0.0:50279] timeout=300.0>) 2021-05-17 03:40:35,717 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff93 type=client open_count=1 channel=[id: 0x706f7bc7, 0.0.0.0/0.0.0.0:34242] timeout=300.0>) 2021-05-17 03:41:01,903 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff94 type=client open_count=1 channel=[id: 0x7e95b8f6, 0.0.0.0/0.0.0.0:38346] timeout=300.0>) 2021-05-17 03:41:03,928 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x67ff95 type=client open_count=1 channel=[id: 0xc79a21ba, 0.0.0.0/0.0.0.0:46609] timeout=300.0>) 2021-05-17 03:41:05,956 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff96 type=client open_count=1 channel=[id: 0x9c6b42a8, 0.0.0.0/0.0.0.0:53922] timeout=300.0>) 2021-05-17 03:41:35,891 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff97 type=client open_count=1 channel=[id: 0x3c3db1e6, 0.0.0.0/0.0.0.0:41188] timeout=300.0>) 2021-05-17 03:41:37,920 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x67ff98 type=client open_count=1 channel=[id: 0x79201f28, 0.0.0.0/0.0.0.0:46867] timeout=300.0>) 2021-05-17 03:41:39,947 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:42:04,108 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:42:20,650 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:42:20,690 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:42:20,762 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:42:34,428 | test | INFO | pool-14-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 3.30666666667 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:42:34,473 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:42:44,595 | test | INFO | pool-14-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:42:44,641 | test | INFO | pool-14-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.2120001316 sec 2021-05-17 03:42:45,061 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:43:32,223 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:43:32,674 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 0 | 1048576000 | 946590824 | 110992556 | | bucket2 | ephemeral | 3 | none | 0 | 0 | 1048576000 | 284335920 | 170 | | default | couchbase | 3 | none | 0 | 0 | 10485760000 | 1380384840 | 51620187 | +---------+-----------+----------+------------+-----+-------+-------------+------------+-----------+ 2021-05-17 03:43:32,676 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 03:44:15,217 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:44:15,227 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:44:36,875 | test | INFO | pool-14-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....................................................................................................*X | 24143.210 | +-------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:44:38,726 | test | INFO | pool-14-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.....................................................................**..............................X | 176.725 | +-------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:44:40,346 | test | INFO | pool-14-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+-------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+-------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***..**..................................................................................................*X | 291.046 | +-------+-------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 03:44:40,601 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.6355246439 | 6437847040 | 4209774592 | 2546814976 / 7511588864 | 100582 / 256259 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.791992545 | 6437847040 | 4168409088 | 2590584832 / 7511588864 | 100135 / 203771 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.8433307222 | 6437847040 | 4163166208 | 2591281152 / 7511588864 | 98983 / 242447 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.9750378592 | 6437847040 | 3679748096 | 3278614528 / 9530753024 | 101211 / 295959 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.7910118371 | 6437847040 | 4147585024 | 2629996544 / 7511588864 | 100565 / 258036 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 03:44:53,809 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 03:46:21,121 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 3 | none | 0 | 3000 | 1048576000 | 212938088 | 293539522 | | bucket2 | ephemeral | 3 | none | 0 | 3000 | 1048576000 | 319628640 | 170 | | default | couchbase | 3 | none | 0 | 500000 | 10485760000 | 836209640 | 604294682 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 03:46:21,121 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #12 test_data_load_collections_with_hard_failover_recovery ========= 2021-05-17 03:46:31,273 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_recovery 2021-05-17 03:46:31,275 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_recovery 2021-05-17 03:46:31,276 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:734] failing over nodes [ip:172.23.107.131 port:8091 ssh_username:Administrator, ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 03:46:43,181 | test | INFO | pool-14-thread-26 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.20299983025 seconds to complete 2021-05-17 03:46:43,187 | test | INFO | pool-14-thread-26 | [common_lib:sleep:22] Sleep 8.20299983025 seconds. Reason: Wait after rebalance complete 2021-05-17 03:48:51,448 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.055999994278 seconds 2021-05-17 03:49:02,905 | test | INFO | pool-14-thread-8 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.24199986458 seconds to complete 2021-05-17 03:49:02,911 | test | INFO | pool-14-thread-8 | [common_lib:sleep:22] Sleep 8.24199986458 seconds. Reason: Wait after rebalance complete 2021-05-17 03:51:11,213 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 2 nodes failed over as expected in 0.058000087738 seconds 2021-05-17 03:51:11,214 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 03:51:11,216 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:51:18,375 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:51:32,852 | test | INFO | pool-14-thread-23 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 11143.727 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:51:34,207 | test | INFO | pool-14-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 3612.957 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:51:34,420 | test | INFO | pool-14-thread-11 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................*X | 2574.382 | +-------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:51:34,528 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,529 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,585 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.131 - Node not part of cluster inactiveFailed 2021-05-17 03:51:34,585 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 03:51:49,523 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 03:51:49,914 | test | INFO | pool-14-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 1.015 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 78.8049339445 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 69.9811380594 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 86.8874659355 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 0.521649278357 | Cluster node | +----------------+----------+-----------------------+----------------+--------------+ 2021-05-17 03:51:50,039 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:51:55,167 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:51:56,000 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 03:52:00,651 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:06,823 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:11,358 | test | INFO | pool-14-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **....................................................................................................X | 13046.518 | +-------+---------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 03:52:11,428 | test | INFO | pool-14-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 5236.475 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:52:11,496 | test | INFO | pool-14-thread-30 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................X | 4891.964 | +-------+---------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 03:52:11,963 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:17,540 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:22,801 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:52:27,875 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.43 2021-05-17 03:52:33,002 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.75 2021-05-17 03:52:38,084 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 5.79 2021-05-17 03:52:43,207 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 9.34 2021-05-17 03:52:48,365 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 14.01 2021-05-17 03:52:53,540 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 18.55 2021-05-17 03:52:58,753 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.46 2021-05-17 03:53:03,884 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 26.67 2021-05-17 03:53:09,023 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 28.98 2021-05-17 03:53:14,197 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 34.87 2021-05-17 03:53:19,299 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.05 2021-05-17 03:53:24,424 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.26 2021-05-17 03:53:29,538 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.19 2021-05-17 03:53:34,690 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 51.81 2021-05-17 03:53:41,391 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 55.4 2021-05-17 03:53:46,532 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 62.56 2021-05-17 03:53:51,642 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.84 2021-05-17 03:53:56,828 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 74.8 2021-05-17 03:54:01,928 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: running, progress: 78.75 2021-05-17 03:54:12,028 | test | INFO | pool-14-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:54:12,068 | test | INFO | pool-14-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 142.154999971 sec 2021-05-17 03:54:13,066 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 03:54:13,068 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 03:54:29,040 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 03:54:29,042 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 03:54:49,084 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 03:54:49,084 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 03:55:41,694 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1229999065 seconds to complete 2021-05-17 03:55:41,710 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 998.305s OK During the test, Remote Connections: 127, Disconnections: 127 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 12 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_12 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_graceful_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 13, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_graceful_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 03:55:51,174 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:55:51,875 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:55:52,308 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183aa type=client open_count=1 channel=[id: 0xc6159f2c, 0.0.0.0/0.0.0.0:50670] timeout=300.0>) 2021-05-17 03:56:13,062 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183ab type=client open_count=1 channel=[id: 0x197aaf49, 0.0.0.0/0.0.0.0:39074] timeout=300.0>) 2021-05-17 03:56:15,078 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7183ac type=client open_count=1 channel=[id: 0xd49c0389, 0.0.0.0/0.0.0.0:42013] timeout=300.0>) 2021-05-17 03:56:17,095 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183ad type=client open_count=1 channel=[id: 0x0ebae3ab, 0.0.0.0/0.0.0.0:47745] timeout=300.0>) 2021-05-17 03:56:41,036 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183ae type=client open_count=1 channel=[id: 0xe01f95b7, 0.0.0.0/0.0.0.0:36990] timeout=300.0>) 2021-05-17 03:56:43,055 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7183af type=client open_count=1 channel=[id: 0xdcd09b19, 0.0.0.0/0.0.0.0:41551] timeout=300.0>) 2021-05-17 03:56:45,072 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b0 type=client open_count=1 channel=[id: 0x79965bac, 0.0.0.0/0.0.0.0:45676] timeout=300.0>) 2021-05-17 03:57:09,621 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b1 type=client open_count=1 channel=[id: 0xae642b6c, 0.0.0.0/0.0.0.0:36085] timeout=300.0>) 2021-05-17 03:57:11,638 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7183b2 type=client open_count=1 channel=[id: 0x825a5e3e, 0.0.0.0/0.0.0.0:55391] timeout=300.0>) 2021-05-17 03:57:13,655 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b3 type=client open_count=1 channel=[id: 0xf0cbba97, 0.0.0.0/0.0.0.0:59638] timeout=300.0>) 2021-05-17 03:57:37,671 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b4 type=client open_count=1 channel=[id: 0x0d4f207a, 0.0.0.0/0.0.0.0:52352] timeout=300.0>) 2021-05-17 03:57:39,691 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7183b5 type=client open_count=1 channel=[id: 0x36382df1, 0.0.0.0/0.0.0.0:36026] timeout=300.0>) 2021-05-17 03:57:41,709 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b6 type=client open_count=1 channel=[id: 0xda666016, 0.0.0.0/0.0.0.0:45077] timeout=300.0>) 2021-05-17 03:58:05,575 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b7 type=client open_count=1 channel=[id: 0xfca74b30, 0.0.0.0/0.0.0.0:58847] timeout=300.0>) 2021-05-17 03:58:07,592 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7183b8 type=client open_count=1 channel=[id: 0xf7b71b6d, 0.0.0.0/0.0.0.0:41192] timeout=300.0>) 2021-05-17 03:58:09,609 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 03:58:26,941 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 03:58:40,677 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 03:58:40,698 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:58:40,734 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 03:58:54,697 | test | INFO | pool-15-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 5.08149728342 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 03:58:54,720 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 03:59:04,792 | test | INFO | pool-15-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 03:59:04,822 | test | INFO | pool-15-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1259999275 sec 2021-05-17 03:59:05,072 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 03:59:50,569 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 03:59:50,859 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 627925672 | 98651537 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 211980088 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 562588744 | 79934784 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 03:59:50,861 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:00:33,388 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:00:33,397 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:01:13,168 | test | INFO | pool-15-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *.........**...........................................................................................*X | 13531.578 | | 13288.278 | .................................................................................................X | 13288.278 | | 13261.156 | .................................................................................................*X | 13261.156 | | 12716.754 | ............................................................................................X | 12716.754 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:01:14,589 | test | INFO | pool-15-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | *.**...**..........................*.**.....................................................................*X | 154.514 | +-------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:01:14,867 | test | INFO | pool-15-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +--------+---------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *...................................................................................**.....**X | 69.625 | | 55.598 | .....................................................................**..............................*X | 78.707 | +--------+---------------------------------------------------------------------------------------------------------+--------+ 2021-05-17 04:01:15,125 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.5526473927 | 6437847040 | 4255506432 | 2479099904 / 7511588864 | 83463 / 165643 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.9488119417 | 6437847040 | 4221444096 | 2515341312 / 7511588864 | 81385 / 162604 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.87 | 6437847040 | 4228124672 | 2502098944 / 7511588864 | 82728 / 167418 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 99.4014278787 | 6437847040 | 3734384640 | 3193126912 / 9530753024 | 83178 / 163179 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.5051160302 | 6437847040 | 4281110528 | 2475749376 / 7511588864 | 82364 / 165807 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:01:28,927 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:02:55,592 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162373384 | 241888316 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 245360528 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 609616504 | 405183413 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:02:55,595 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #13 test_data_load_collections_with_graceful_failover_rebalance_out ========= 2021-05-17 04:03:05,671 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during graceful_failover_rebalance_out 2021-05-17 04:03:05,673 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : graceful_failover_rebalance_out 2021-05-17 04:03:05,674 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:585] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:03:48,188 | test | INFO | pool-15-thread-22 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 42.3129999638 seconds to complete 2021-05-17 04:03:48,194 | test | INFO | pool-15-thread-22 | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete 2021-05-17 04:05:53,250 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0529999732971 seconds 2021-05-17 04:05:53,250 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:05:53,252 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:05:59,601 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:06:31,973 | test | INFO | pool-15-thread-29 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **.........................................**..............*..........................................**...*X | 12877.411 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:06:33,095 | test | INFO | pool-15-thread-20 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +--------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*..................................**........................................................**X | 2125.554 | | 46.009 | .*X | 46.009 | +--------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:06:34,223 | test | INFO | pool-15-thread-28 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *..........*.............................**.............................................................**X | 2413.283 | | 234.050 | ........**X | 234.050 | +---------+-------------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:06:34,470 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:06:34,526 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:06:45,240 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:06:45,415 | test | INFO | pool-15-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 4.92016400547 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 99.4000099998 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 51.1217199706 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 59.5319843995 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 52.7074548758 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 04:06:45,453 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:06:51,194 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:06:52,068 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:06:54,147 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,148 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,155 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,157 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,157 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-381 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:54,158 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-211 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-381 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:55,999 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,000 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,160 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,161 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,171 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,173 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1678 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,408 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1539 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,411 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1539 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:56,437 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 1.46 2021-05-17 04:06:57,918 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3356 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:57,920 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3356 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,546 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3157 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,548 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3157 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,851 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3364 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,852 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3364 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,993 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:58,994 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,003 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3562 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,005 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3562 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,263 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,265 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,364 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3210 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,365 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3380 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3210 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,372 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,374 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,424 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,426 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3625 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,733 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,734 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,740 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,740 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,765 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:06:59,766 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3834 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,045 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,046 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,049 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,049 | test | INFO | pool-15-thread-17 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,053 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4133 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4035 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,055 | test | INFO | pool-15-thread-14 | [table_view:display:72] Keys failed in bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4133 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4035 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,329 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:00,331 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4052 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:01,663 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.49 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 10 docs from 0 to 5000 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 6 docs from 0 to 5000 2021-05-17 04:07:02,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 10 docs from 0 to 5000 2021-05-17 04:07:02,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1249 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1266 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:02,885 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1514 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1484 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:03,091 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1645 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:03,588 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 8 docs from 0 to 5000 2021-05-17 04:07:04,305 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-395 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,305 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-205 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-395 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,378 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1279 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,378 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1279 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,644 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-577 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:04,644 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-577 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,178 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-897 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,180 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-907 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-897 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,792 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2419 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:05,793 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2589 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2419 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,128 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1646 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1638 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,130 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1646 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1638 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,275 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2963 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2800 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2990 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,276 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2963 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2800 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2990 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,543 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3094 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3067 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3104 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,545 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3094 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3067 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3104 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:06,782 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 3.52 2021-05-17 04:07:07,346 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3602 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3792 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:07,348 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3602 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3792 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,252 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2832 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,252 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2832 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,674 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,676 | test | INFO | pool-15-thread-28 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4643 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:08,868 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 12 docs from 0 to 5000 2021-05-17 04:07:08,872 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 5 docs from 1000 to 1960 2021-05-17 04:07:09,441 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3949 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3915 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-3885 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,443 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3949 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3915 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3893 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-3885 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,710 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4187 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4017 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:09,711 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4187 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4017 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,267 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4673 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,269 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4673 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4838 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | INFO | pool-15-thread-8 | [table_view:display:72] Keys failed in bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4838 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:10,559 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 19 docs from 0 to 5000 2021-05-17 04:07:12,111 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 4.55 2021-05-17 04:07:14,490 | test | INFO | pool-15-thread-13 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:awM3zYY-57-714000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1518 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1488 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:15,204 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 04:07:16,588 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,588 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-335 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,822 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-447 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,822 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-447 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,849 | test | INFO | pool-15-thread-4 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:awM3zYY-57-714000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Delete Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-318 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-335 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | | test_collections-288 | {'error': 'com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible', 'cas': 0, 'value': {}} | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,924 | test | INFO | pool-15-thread-21 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5635 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5618 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-5788 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:16,926 | test | INFO | pool-15-thread-21 | [table_view:display:72] Keys failed in bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-5635 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5618 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-5788 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:17,260 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 5.86 2021-05-17 04:07:18,556 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:18,558 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1059 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:19,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2452 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:19,677 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2452 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,632 | test | INFO | pool-15-thread-12 | [table_view:display:72] Keys failed in bucket1:K9hJ4pRnsUnNqO%q-57-888000:kur163WM_Tne-57-990000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1741 | com.couchbase.client.core.error.DurabilityAmbiguousException | The server returned with a durability ambiguous response on this request | | test_collections-1626 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,671 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4045 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,671 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4045 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,913 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4219 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4389 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:21,915 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4219 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4389 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,285 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4617 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4787 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,286 | test | INFO | pool-15-thread-16 | [table_view:display:72] Keys failed in bucket1:scope1:mHv6XNug-51-818000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4617 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4787 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:22,427 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 9 docs from 0 to 5000 2021-05-17 04:07:22,430 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 3 docs from 5000 to 5800 2021-05-17 04:07:22,431 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 3 docs from 0 to 1000 2021-05-17 04:07:22,443 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 2 docs from 1000 to 1960 2021-05-17 04:07:22,553 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 6.94 2021-05-17 04:07:24,421 | test | INFO | pool-15-thread-7 | [table_view:display:72] Keys failed in bucket1:scope2:l8lbEc7j62ZBQ7qJ-5-134000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:24,701 | test | INFO | pool-15-thread-27 | [table_view:display:72] Keys failed in bucket1:_default:ryf37NkjR9%G-5-128000 +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Update Key | Exception | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-6 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +--------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:07:25,785 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 04:07:25,911 | test | ERROR | pool-15-thread-2 | [task:call:3809] Failed to load 1 docs from 5 to 7 2021-05-17 04:07:27,846 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 8.36 2021-05-17 04:07:29,107 | test | INFO | pool-15-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *...................**.................................................................................*X | 11691.578 | | 10062.500 | .....................................................................................*X | 10062.500 | | 9391.399 | ...............................................................................X | 9391.399 | +-----------+-----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:07:29,236 | test | INFO | pool-15-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *............**........................................................................................X | 2489.068 | | 2489.068 | ..................................................................................................X | 2489.068 | | 1146.803 | .............................................*X | 1146.803 | | 1083.434 | ..........................................X | 1083.434 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:07:31,223 | test | INFO | pool-15-thread-25 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | *.......**.............................................................................................*X | 2583.773 | | 2209.022 | ....................................................................................*X | 2209.022 | | 1337.462 | ..................................................X | 1337.462 | | 1028.967 | ......................................X | 1028.967 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:07:32,948 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 11.28 2021-05-17 04:07:38,055 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 14.54 2021-05-17 04:07:43,131 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 19.7 2021-05-17 04:07:48,230 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 26.7 2021-05-17 04:07:53,313 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-17 04:07:58,384 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.06 2021-05-17 04:08:03,500 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 49.73 2021-05-17 04:08:08,611 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 59.66 2021-05-17 04:08:13,973 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 66.78 2021-05-17 04:08:19,088 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 74.32 2021-05-17 04:08:24,200 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 81.93 2021-05-17 04:08:29,285 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 89.58 2021-05-17 04:08:34,391 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: running, progress: 96.85 2021-05-17 04:08:44,493 | test | INFO | pool-15-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:08:44,625 | test | INFO | pool-15-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 119.209000111 sec 2021-05-17 04:08:44,673 | test | WARNING | Thread-43449 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:_default:ryf37NkjR9%G-5-128000 update failed keys from task: [u'test_collections-6'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43453 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:JQl%ePWM-49-690000:0JPxQxKFwwWE_lOo-49-822000 create failed keys from task: [u'test_collections-3380', u'test_collections-4052', u'test_collections-211', u'test_collections-1678', u'test_collections-381', u'test_collections-3210'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43454 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:JQl%ePWM-49-690000:SpzGyZAkO-49-823000 create failed keys from task: [u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-1539', u'test_collections-4133', u'test_collections-4035', u'test_collections-3364', u'test_collections-3157'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43455 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8RfzGN%917RjOPHkUfX2-50-497000:s-h4o96z-50-609000 create failed keys from task: [u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-211', u'test_collections-1678', u'test_collections-4133', u'test_collections-381', u'test_collections-4035', u'test_collections-3562', u'test_collections-3356'] 2021-05-17 04:08:44,674 | test | WARNING | Thread-43456 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:8RfzGN%917RjOPHkUfX2-50-497000:EbkElNT-50-608000 create failed keys from task: [u'test_collections-3380', u'test_collections-4052', u'test_collections-3625', u'test_collections-3834', u'test_collections-211', u'test_collections-1678', u'test_collections-4133', u'test_collections-381', u'test_collections-4035', u'test_collections-3210'] 2021-05-17 04:08:44,677 | test | WARNING | Thread-43459 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:kE%xlFqkQjNXvfV7iB-57-713000 create failed keys from task: [u'test_collections-5635', u'test_collections-5618', u'test_collections-5788'] 2021-05-17 04:08:44,677 | test | WARNING | Thread-43458 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:K9hJ4pRnsUnNqO%q-57-888000:kur163WM_Tne-57-990000 update failed keys from task: [u'test_collections-1741', u'test_collections-1626'] 2021-05-17 04:08:44,687 | test | WARNING | Thread-43457 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:K9hJ4pRnsUnNqO%q-57-888000:N0yLPHzgHgfXybMgT-57-989000 update failed keys from task: [u'test_collections-1249', u'test_collections-1645', u'test_collections-1514', u'test_collections-1484', u'test_collections-1266'] 2021-05-17 04:08:44,721 | test | WARNING | Thread-43460 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:awM3zYY-57-714000 update failed keys from task: [u'test_collections-1518', u'test_collections-1488'] 2021-05-17 04:08:44,743 | test | WARNING | Thread-43460 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:diobobWdr-57-599000:awM3zYY-57-714000 delete failed keys from task: [u'test_collections-318', u'test_collections-335', u'test_collections-288'] 2021-05-17 04:08:44,773 | test | WARNING | Thread-43486 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope1:mHv6XNug-51-818000 create failed keys from task: [u'test_collections-447', u'test_collections-335', u'test_collections-1059', u'test_collections-4219', u'test_collections-4617', u'test_collections-4045', u'test_collections-2452', u'test_collections-4787', u'test_collections-4389'] 2021-05-17 04:08:44,789 | test | WARNING | Thread-43507 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:scope2:l8lbEc7j62ZBQ7qJ-5-134000 update failed keys from task: [u'test_collections-6'] 2021-05-17 04:08:44,792 | test | WARNING | Thread-43529 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rfSPLc2%QQu-50-26000:l6Kob7C7L-50-156000 create failed keys from task: [u'test_collections-3094', u'test_collections-3602', u'test_collections-2419', u'test_collections-2990', u'test_collections-3067', u'test_collections-3104', u'test_collections-1279', u'test_collections-2589', u'test_collections-2963', u'test_collections-2800', u'test_collections-3792', u'test_collections-4643'] 2021-05-17 04:08:44,792 | test | WARNING | Thread-43528 | [bucket_ready_functions:validate_crud_task_per_collection:501] bucket1:rfSPLc2%QQu-50-26000:Ou9n34twnbI-50-157000 create failed keys from task: [u'test_collections-4082', u'test_collections-907', u'test_collections-3949', u'test_collections-205', u'test_collections-2832', u'test_collections-1646', u'test_collections-4838', u'test_collections-3903', u'test_collections-4112', u'test_collections-4673', u'test_collections-4017', u'test_collections-3885', u'test_collections-577', u'test_collections-897', u'test_collections-395', u'test_collections-3915', u'test_collections-1638', u'test_collections-4187', u'test_collections-3893'] 2021-05-17 04:08:55,312 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:09:58,482 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket2, Active Resident ratio(DGM): 100% 2021-05-17 04:09:58,482 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket2, Replica Resident ratio(DGM): 100% 2021-05-17 04:10:10,967 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: bucket1, Active Resident ratio(DGM): 100% 2021-05-17 04:10:10,967 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: bucket1, Replica Resident ratio(DGM): 100% 2021-05-17 04:10:26,183 | test | INFO | MainThread | [collections_base:tearDown:99] Bucket: default, Active Resident ratio(DGM): 100% 2021-05-17 04:10:26,184 | test | INFO | MainThread | [collections_base:tearDown:102] Bucket: default, Replica Resident ratio(DGM): 100% 2021-05-17 04:11:19,997 | test | INFO | MainThread | [rest_client:monitorRebalance:1484] Rebalance done. Taken 11.1740000248 seconds to complete 2021-05-17 04:11:20,013 | test | INFO | MainThread | [common_lib:sleep:22] Sleep 5 seconds. Reason: Wait after rebalance complete ok ---------------------------------------------------------------------- Ran 1 test in 938.230s OK During the test, Remote Connections: 190, Disconnections: 190 SDK Connections: 30, Disconnections: 30 summary so far suite bucket_collections.collections_rebalance.CollectionsRebalance , pass 13 , fail 0 testrunner logs, diags and results are available under /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_13 Logs will be stored at /data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_14 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/testexec-temp_win_private_job_3.ini rerun=False,get-cbcollect-info=True,quota_percent=95,crash_warning=True -t bucket_collections.collections_rebalance.CollectionsRebalance.test_data_load_collections_with_hard_failover_rebalance_out,nodes_init=5,nodes_failover=1,override_spec_params=durability;replicas,durability=MAJORITY,replicas=2,bucket_spec=multi_bucket.buckets_for_rebalance_tests_more_collections,data_load_spec=volume_test_load_with_CRUD_on_collections,data_load_stage=during,quota_percent=80,skip_validations=False,GROUP=failover_with_collection_crud_durability_MAJORITY' Test Input params: {'data_load_stage': 'during', 'cluster_name': 'testexec-temp_win_private_job_3', 'quota_percent': '95', 'nodes_failover': '1', 'ini': '/tmp/testexec-temp_win_private_job_3.ini', 'get-cbcollect-info': 'True', 'replicas': '2', 'durability': 'MAJORITY', 'conf_file': 'conf/collections/collections_failover_crud_on_collections.conf', 'crash_warning': 'True', 'skip_validations': 'False', 'spec': 'collections_failover_crud_on_collections', 'rerun': 'False', 'num_nodes': 5, 'logs_folder': '/data/workspace/temp_win_private_job_3/logs/testrunner-21-May-17_00-18-12/test_14', 'nodes_init': '5', 'GROUP': 'failover_with_collection_crud_durability_MAJORITY', 'bucket_spec': 'multi_bucket.buckets_for_rebalance_tests_more_collections', 'case_number': 14, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections', 'override_spec_params': 'durability;replicas'} test_data_load_collections_with_hard_failover_rebalance_out (bucket_collections.collections_rebalance.CollectionsRebalance) ... 2021-05-17 04:11:29,443 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionsRebalance setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:11:30,111 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:11:30,529 | test | INFO | MainThread | [basetestcase:setUp:321] Initializing cluster Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c65 type=client open_count=1 channel=[id: 0x3e52663d, 0.0.0.0/0.0.0.0:38511] timeout=300.0>) 2021-05-17 04:11:51,305 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c66 type=client open_count=1 channel=[id: 0xe3f7f34f, 0.0.0.0/0.0.0.0:41511] timeout=300.0>) 2021-05-17 04:11:53,322 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.126:8091 during connect (<_realsocket at 0x7b2c67 type=client open_count=1 channel=[id: 0x43a8daa1, 0.0.0.0/0.0.0.0:37679] timeout=300.0>) 2021-05-17 04:11:55,341 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.126:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c68 type=client open_count=1 channel=[id: 0x2dc8edfc, 0.0.0.0/0.0.0.0:39028] timeout=300.0>) 2021-05-17 04:12:19,454 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c69 type=client open_count=1 channel=[id: 0x853a4dba, 0.0.0.0/0.0.0.0:40667] timeout=300.0>) 2021-05-17 04:12:21,473 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x7b2c6a type=client open_count=1 channel=[id: 0xd34949ed, 0.0.0.0/0.0.0.0:58698] timeout=300.0>) 2021-05-17 04:12:23,490 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6b type=client open_count=1 channel=[id: 0x859eecf7, 0.0.0.0/0.0.0.0:39930] timeout=300.0>) 2021-05-17 04:12:47,519 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6c type=client open_count=1 channel=[id: 0xec866551, 0.0.0.0/0.0.0.0:54312] timeout=300.0>) 2021-05-17 04:12:49,538 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.129:8091 during connect (<_realsocket at 0x7b2c6d type=client open_count=1 channel=[id: 0xf0e181ed, 0.0.0.0/0.0.0.0:36732] timeout=300.0>) 2021-05-17 04:12:51,555 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.129:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c6e type=client open_count=1 channel=[id: 0x153579a7, 0.0.0.0/0.0.0.0:50064] timeout=300.0>) 2021-05-17 04:13:15,821 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c6f type=client open_count=1 channel=[id: 0x74ef2bb5, 0.0.0.0/0.0.0.0:33732] timeout=300.0>) 2021-05-17 04:13:17,838 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0x7b2c70 type=client open_count=1 channel=[id: 0x8506e7ab, 0.0.0.0/0.0.0.0:60279] timeout=300.0>) 2021-05-17 04:13:19,858 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c71 type=client open_count=1 channel=[id: 0xd5343ac4, 0.0.0.0/0.0.0.0:55888] timeout=300.0>) 2021-05-17 04:13:44,591 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c72 type=client open_count=1 channel=[id: 0x411b3cc5, 0.0.0.0/0.0.0.0:41852] timeout=300.0>) 2021-05-17 04:13:46,608 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c73 type=client open_count=1 channel=[id: 0xf086346e, 0.0.0.0/0.0.0.0:38017] timeout=300.0>) 2021-05-17 04:13:48,627 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.107.142:8091 during connect (<_realsocket at 0x7b2c74 type=client open_count=1 channel=[id: 0x1d0d6ea6, 0.0.0.0/0.0.0.0:39965] timeout=300.0>) 2021-05-17 04:13:50,644 | test | ERROR | MainThread | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.142:8091/nodes/self error [Errno 111] Connection refused 2021-05-17 04:14:07,154 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:596] RAM quota was defined less than 100 MB: 2021-05-17 04:14:20,881 | test | INFO | MainThread | [basetestcase:setUp:343] Couchbase Cluster: C1, Nodes: 172.23.107.126, 172.23.107.127, 172.23.107.129, 172.23.107.131, 172.23.107.142 initialized 2021-05-17 04:14:20,903 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= BaseTestCase setup finished for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:14:20,941 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup started for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:14:35,917 | test | INFO | pool-16-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+--------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+--------------+--------------+ | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 4.8434140569 | Cluster node | | 172.23.107.127 | None | | | <--- IN --- | | 172.23.107.129 | None | | | <--- IN --- | | 172.23.107.131 | None | | | <--- IN --- | | 172.23.107.142 | None | | | <--- IN --- | +----------------+----------+-----------------------+--------------+--------------+ 2021-05-17 04:14:35,941 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:14:46,013 | test | INFO | pool-16-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-17 04:14:46,040 | test | INFO | pool-16-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 10.1239998341 sec 2021-05-17 04:14:46,286 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1799] Creating required buckets from template 2021-05-17 04:15:31,529 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2260] Waiting for all collections to be created 2021-05-17 04:15:31,792 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 0 | 1048576000 | 627715080 | 117765415 | | bucket2 | ephemeral | 2 | none | 0 | 0 | 1048576000 | 225369296 | 170 | | default | couchbase | 2 | none | 0 | 0 | 10485760000 | 530472984 | 103447262 | +---------+-----------+----------+------------+-----+-------+-------------+-----------+-----------+ 2021-05-17 04:15:31,792 | test | INFO | MainThread | [collections_base:collection_setup:175] Creating required SDK clients for client_pool 2021-05-17 04:16:16,030 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:16:16,063 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:16:58,023 | test | INFO | pool-16-thread-11 | [table_view:display:72] Ops trend for bucket 'default' +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.........................................................................................**..........**.*X | 12351.784 | +-------+---------------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:16:58,170 | test | INFO | pool-16-thread-12 | [table_view:display:72] Ops trend for bucket 'bucket1' +--------+-----------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +--------+-----------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***....................................................................................................*X | 182.713 | | 25.471 | ............**.............*X | 48.779 | +--------+-----------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:16:59,586 | test | INFO | pool-16-thread-13 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+----------------------------------------------------------------------------------------------------------------+---------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------+---------+ | 0.000 | ***.......**............**...........................................**......................................X | 157.407 | +-------+----------------------------------------------------------------------------------------------------------------+---------+ 2021-05-17 04:16:59,931 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ | 172.23.107.142 | kv | 99.7109154345 | 6437847040 | 4352671744 | 2448977920 / 7511588864 | 97757 / 196365 | 7.0.0-5172-enterprise | | 172.23.107.127 | kv | 99.4808047526 | 6437847040 | 4240502784 | 2496221184 / 7511588864 | 96182 / 192500 | 7.0.0-5172-enterprise | | 172.23.107.129 | kv | 99.0318649952 | 6437847040 | 4221915136 | 2504466432 / 7511588864 | 96611 / 194324 | 7.0.0-5172-enterprise | | 172.23.107.126 | kv | 100 | 6437847040 | 3840958464 | 3076243456 / 9530753024 | 96866 / 191250 | 7.0.0-5172-enterprise | | 172.23.107.131 | kv | 99.3716980818 | 6437847040 | 4255543296 | 2492289024 / 7511588864 | 96906 / 191023 | 7.0.0-5172-enterprise | +----------------+----------+-----------------+------------+------------+-------------------------+-------------------+-----------------------+ 2021-05-17 04:17:14,092 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4406] Validating collection stats and item counts 2021-05-17 04:18:40,354 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ | bucket1 | couchbase | 2 | none | 0 | 3000 | 1048576000 | 162403336 | 236910036 | | bucket2 | ephemeral | 2 | none | 0 | 3000 | 1048576000 | 246185040 | 170 | | default | couchbase | 2 | none | 0 | 500000 | 10485760000 | 544550472 | 406719618 | +---------+-----------+----------+------------+-----+--------+-------------+-----------+-----------+ 2021-05-17 04:18:40,355 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= CollectionBase setup complete for test #14 test_data_load_collections_with_hard_failover_rebalance_out ========= 2021-05-17 04:18:50,703 | test | INFO | MainThread | [collections_rebalance:load_collections_with_rebalance:907] Doing collection data load during hard_failover_rebalance_out 2021-05-17 04:18:50,703 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:388] Starting rebalance operation of type : hard_failover_rebalance_out 2021-05-17 04:18:50,704 | test | INFO | MainThread | [collections_rebalance:rebalance_operation:632] failing over nodes [ip:172.23.107.142 port:8091 ssh_username:Administrator] 2021-05-17 04:19:01,677 | test | INFO | pool-16-thread-1 | [rest_client:monitorRebalance:1484] Rebalance done. Taken 8.3789999485 seconds to complete 2021-05-17 04:19:01,683 | test | INFO | pool-16-thread-1 | [common_lib:sleep:22] Sleep 8.3789999485 seconds. Reason: Wait after rebalance complete 2021-05-17 04:21:10,121 | test | INFO | MainThread | [collections_rebalance:wait_for_failover_or_assert:358] 1 nodes failed over as expected in 0.0549998283386 seconds 2021-05-17 04:21:10,121 | test | INFO | MainThread | [collections_rebalance:data_load_after_failover:337] Starting a sync data load after failover 2021-05-17 04:21:10,121 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:21:16,464 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:21:47,653 | test | INFO | pool-16-thread-3 | [table_view:display:72] Ops trend for bucket 'default' +----------+----------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***.......................................................................X | 9601.421 | | 9601.421 | ......................................................................**.............................**X | 13339.375 | +----------+----------------------------------------------------------------------------------------------------------+-----------+ 2021-05-17 04:21:48,930 | test | INFO | pool-16-thread-8 | [table_view:display:72] Ops trend for bucket 'bucket1' +----------+----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **...................................................................................................**X | 1728.282 | | 1340.883 | ............................................................................**X | 1340.883 | | 56.522 | ..X | 56.522 | +----------+----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:21:49,555 | test | INFO | pool-16-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket2' +----------+-----------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +----------+-----------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **....................................................................................................**X | 2222.753 | | 1352.913 | ...........................................................**X | 1352.913 | | 45.335 | .*X | 45.335 | +----------+-----------------------------------------------------------------------------------------------------------+----------+ 2021-05-17 04:21:49,832 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:21:49,884 | test | WARNING | MainThread | [rest_client:get_nodes:1756] 172.23.107.142 - Node not part of cluster inactiveFailed 2021-05-17 04:22:01,581 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4652] Performing scope/collection specific operations 2021-05-17 04:22:01,786 | test | INFO | pool-16-thread-7 | [table_view:display:72] Rebalance Overview +----------------+----------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5172-enterprise | 8.51819136348 | --- OUT ---> | | 172.23.107.127 | kv | 7.0.0-5172-enterprise | 85.5523653724 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5172-enterprise | 86.5970074396 | Cluster node | | 172.23.107.126 | kv | 7.0.0-5172-enterprise | 91.9657463359 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5172-enterprise | 82.5512878295 | Cluster node | +----------------+----------+-----------------------+---------------+--------------+ 2021-05-17 04:22:01,819 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:22:08,605 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-17 04:22:08,770 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4742] Done Performing scope/collection specific operations 2021-05-17 04:22:13,562 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2076 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:13,566 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2076 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:13,780 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 1.61 2021-05-17 04:22:15,223 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-171 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,223 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-171 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,309 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3138 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,311 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3138 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,424 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:15,426 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-120 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,358 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1022 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1198 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,361 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1022 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1198 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,611 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4119 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4089 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,611 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4119 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4089 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,617 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1198 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,618 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1008 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1198 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,651 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,651 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,808 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-32 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +---------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,809 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-32 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +---------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,861 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,862 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4261 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,871 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,872 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1370 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,888 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1402 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1592 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:16,888 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1402 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1592 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,398 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,400 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,983 | test | INFO | pool-16-thread-18 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:K6pGTLfDruZK3Z-8-517000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4903 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4893 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4940 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:17,984 | test | INFO | pool-16-thread-18 | [table_view:display:72] Keys failed in bucket1:WTBAjv-14-882000:K6pGTLfDruZK3Z-8-517000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4903 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4893 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4940 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,431 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1812 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1982 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,431 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1812 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1982 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,482 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,483 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2822 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,730 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-2112 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,730 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2082 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-2112 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,733 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-4883 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,734 | test | INFO | pool-16-thread-17 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:6o29bpuptn-oRFx2SD-7-99000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-4913 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-4883 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,736 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3126 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,737 | test | INFO | pool-16-thread-3 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:AbDGSj5y-7-436000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-3126 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,780 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,782 | test | INFO | pool-16-thread-16 | [table_view:display:72] Keys failed in bucket1:kRizd7Dspadfn-6-915000:QvHJj8Tq0-9fzZ-mMr-7-100000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,799 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | | test_collections-1992 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:18,805 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-1802 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | | test_collections-1992 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,108 | test | INFO | pool-16-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.52 2021-05-17 04:22:19,157 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,158 | test | INFO | pool-16-thread-24 | [table_view:display:72] Keys failed in bucket1:PU9QStCCbVSQlpL9-7-247000:soB4WNpiwZ4y-7-437000 +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-2274 | com.couchbase.client.core.error.DurabilityImpossibleException | With the current cluster configuration, the requested durability guarantees are impossible | +-----------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-05-17 04:22:19,171 | test | INFO | pool-16-thread-25 | [table_view:display:72] Keys failed in bucket1:SNXf%C_142QN-6-481000:M_UpxIlVLU-6-750000 +-----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exc