Started by user Sumedh Basarkod [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (P0_win_upgrade jython_slave) in workspace /data/workspace/temp_durability_volume [WS-CLEANUP] Deleting project workspace... Cloning the remote Git repository Cloning repository git://github.com/couchbaselabs/TAF/ > /usr/bin/git init /data/workspace/temp_durability_volume # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url git://github.com/couchbaselabs/TAF/ # timeout=10 Fetching upstream changes from git://github.com/couchbaselabs/TAF/ > /usr/bin/git fetch --tags --progress git://github.com/couchbaselabs/TAF/ +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse refs/remotes/origin/master^{commit} # timeout=10 > /usr/bin/git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision f194813e63e8a1931991aefdee8b10dbf55b072a (refs/remotes/origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f f194813e63e8a1931991aefdee8b10dbf55b072a > /usr/bin/git rev-list f194813e63e8a1931991aefdee8b10dbf55b072a # timeout=10 [temp_durability_volume] $ /bin/sh -xe /tmp/jenkins7755886238252022813.sh + echo '[global] username:root password:couchbase port:8091 n1ql_port:8093 index_port:9102 data_path:/data/ [membase] rest_username:Administrator rest_password:password [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 [_1] ip:172.23.105.175 [_2] ip:172.23.106.233 [_3] ip:172.23.106.236 [_4] ip:172.23.106.238 [_5] ip:172.23.106.250 [_6] ip:172.23.106.251 [_7] ip:172.23.121.74 [_8] ip:172.23.121.78' + git clone https://github.com/sumedhpb/guides.git Cloning into 'guides'... + cp guides/gradle.properties . [temp_durability_volume] $ /bin/sh -xe /tmp/jenkins7635609200498333137.sh + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Cleaning up... + guides/gradlew --refresh-dependencies install -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True' Starting a Gradle Daemon, 67 busy Daemons could not be reused, use --status for details > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -p version=7.0.0-4342,product=cb,parallel=True > Task :install 2021-01-30 02:34:38,926 - root - INFO - Done with uninstall on 172.23.121.74. 2021-01-30 02:34:39,362 - root - INFO - Done with uninstall on 172.23.106.233. 2021-01-30 02:34:43,996 - root - INFO - Done with uninstall on 172.23.106.250. 2021-01-30 02:34:44,108 - root - INFO - Done with uninstall on 172.23.106.236. 2021-01-30 02:34:44,180 - root - INFO - Done with uninstall on 172.23.121.78. 2021-01-30 02:34:44,253 - root - INFO - Done with uninstall on 172.23.106.251. 2021-01-30 02:34:44,362 - root - INFO - Done with uninstall on 172.23.106.238. 2021-01-30 02:34:44,392 - root - INFO - Done with uninstall on 172.23.105.175. 2021-01-30 02:39:20,670 - root - INFO - Done with install on 172.23.106.250. 2021-01-30 02:39:21,536 - root - INFO - Done with install on 172.23.106.238. 2021-01-30 02:39:21,845 - root - INFO - Done with install on 172.23.106.236. 2021-01-30 02:39:23,147 - root - INFO - Done with install on 172.23.105.175. 2021-01-30 02:39:30,573 - root - INFO - Done with install on 172.23.121.74. 2021-01-30 02:39:35,832 - root - INFO - Done with install on 172.23.121.78. 2021-01-30 02:39:36,742 - root - INFO - Done with install on 172.23.106.233. 2021-01-30 02:39:40,357 - root - INFO - Done with install on 172.23.106.251. 2021-01-30 02:40:53,914 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.250 2021-01-30 02:40:53,914 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.250 2021-01-30 02:40:54,075 - root - INFO - Done with init on 172.23.106.250. 2021-01-30 02:40:54,107 - root - INFO - Done with cleanup on 172.23.106.250. 2021-01-30 02:40:54,401 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.238 2021-01-30 02:40:54,403 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.238 2021-01-30 02:40:54,575 - root - INFO - Done with init on 172.23.106.238. 2021-01-30 02:40:54,607 - root - INFO - Done with cleanup on 172.23.106.238. 2021-01-30 02:40:54,733 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.236 2021-01-30 02:40:54,734 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.236 2021-01-30 02:40:54,911 - root - INFO - Done with init on 172.23.106.236. 2021-01-30 02:40:54,940 - root - INFO - Done with cleanup on 172.23.106.236. 2021-01-30 02:40:56,121 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.105.175 2021-01-30 02:40:56,121 - root - INFO - Setting KV memory quota as 256 MB on 172.23.105.175 2021-01-30 02:40:56,286 - root - INFO - Done with init on 172.23.105.175. 2021-01-30 02:40:56,319 - root - INFO - Done with cleanup on 172.23.105.175. 2021-01-30 02:41:02,858 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.74 2021-01-30 02:41:02,858 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.74 2021-01-30 02:41:03,013 - root - INFO - Done with init on 172.23.121.74. 2021-01-30 02:41:03,038 - root - INFO - Done with cleanup on 172.23.121.74. 2021-01-30 02:41:08,187 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.121.78 2021-01-30 02:41:08,187 - root - INFO - Setting KV memory quota as 256 MB on 172.23.121.78 2021-01-30 02:41:08,326 - root - INFO - Done with init on 172.23.121.78. 2021-01-30 02:41:08,348 - root - INFO - Done with cleanup on 172.23.121.78. 2021-01-30 02:41:09,384 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.233 2021-01-30 02:41:09,384 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.233 2021-01-30 02:41:09,519 - root - INFO - Done with init on 172.23.106.233. 2021-01-30 02:41:09,540 - root - INFO - Done with cleanup on 172.23.106.233. 2021-01-30 02:41:13,223 - root - WARNING - KV memory quota is 0MB but needs to be at least 256MB on 172.23.106.251 2021-01-30 02:41:13,224 - root - INFO - Setting KV memory quota as 256 MB on 172.23.106.251 2021-01-30 02:41:13,371 - root - INFO - Done with init on 172.23.106.251. 2021-01-30 02:41:13,391 - root - INFO - Done with cleanup on 172.23.106.251. 2021-01-30 02:41:24,197 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-01-30 02:41:24,221 - root - INFO - node:172.23.105.175:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,246 - root - INFO - node:172.23.106.233:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,266 - root - INFO - node:172.23.106.236:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,288 - root - INFO - node:172.23.106.238:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,309 - root - INFO - node:172.23.106.250:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,332 - root - INFO - node:172.23.106.251:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,355 - root - INFO - node:172.23.121.74:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,391 - root - INFO - node:172.23.121.78:8091 version:7.0.0-4342-enterprise aFamily:inet services:[u'kv'] 2021-01-30 02:41:24,392 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-01-30 02:41:24,394 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-01-30 02:41:24,394 - root - INFO - INSTALL COMPLETED ON: 172.23.105.175 2021-01-30 02:41:24,394 - root - INFO - INSTALL COMPLETED ON: 172.23.106.233 2021-01-30 02:41:24,395 - root - INFO - INSTALL COMPLETED ON: 172.23.106.236 2021-01-30 02:41:24,395 - root - INFO - INSTALL COMPLETED ON: 172.23.106.238 2021-01-30 02:41:24,397 - root - INFO - INSTALL COMPLETED ON: 172.23.106.250 2021-01-30 02:41:24,398 - root - INFO - INSTALL COMPLETED ON: 172.23.106.251 2021-01-30 02:41:24,398 - root - INFO - INSTALL COMPLETED ON: 172.23.121.74 2021-01-30 02:41:24,398 - root - INFO - INSTALL COMPLETED ON: 172.23.121.78 2021-01-30 02:41:24,400 - root - INFO - ---------------------------------------------------------------------------------------------------- 2021-01-30 02:41:24,404 - root - INFO - TOTAL INSTALL TIME = 470.0 seconds Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 8m 2s 1 actionable task: 1 executed + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest' > Configure project : Using Transaction_client :: 1.1.0 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_durability_volume/build/classes/java/main:/data/workspace/temp_durability_volume/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.0/a295271b66684b05dd5345d7b7a6232e03054ef9/couchbase-transactions-1.1.0.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.0.7/892363ab817451b9bb8fa91861c0d66c480056eb/java-client-3.0.7.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.0.8/d21bea503f8be1ce9d68143e4aaf2f203a48d2b7/core-io-2.0.8.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.3.8.RELEASE/e6b2e8dc1f6548ae216b2ecdee0c0a32176317a/reactor-core-3.3.8.RELEASE.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False -m rest > Task :compileJava Note: /data/workspace/temp_durability_volume/src/main/java/com/couchbase/test/transactions/SimpleTransaction.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Global Test input params: {'cluster_name': 'durability_volume', 'ini': '/tmp/durability_volume.ini', 'num_nodes': 8} Logs will be stored at /data/workspace/temp_durability_volume/logs/testrunner-21-Jan-30_02-41-38/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/durability_volume.ini -t volumetests.Collections.volume.test_volume_taf,nodes_init=4,bucket_spec=volume_templates.buckets_for_volume_test,data_load_spec=volume_test_load_with_CRUD_on_collections,iterations=1,quota_percent=100,rerun=False,get-cbcollect-info=True,skip_validations=False' Test Input params: {'cluster_name': 'durability_volume', 'quota_percent': '100', 'get-cbcollect-info': 'True', 'ini': '/tmp/durability_volume.ini', 'iterations': '1', 'rerun': 'False', 'skip_validations': 'False', 'num_nodes': 8, 'logs_folder': '/data/workspace/temp_durability_volume/logs/testrunner-21-Jan-30_02-41-38/test_1', 'nodes_init': '4', 'bucket_spec': 'volume_templates.buckets_for_volume_test', 'case_number': 1, 'data_load_spec': 'volume_test_load_with_CRUD_on_collections'} test_volume_taf (volumetests.Collections.volume) ... 2021-01-30 02:41:47,006 | test | INFO | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #1 test_volume_taf ========= 2021-01-30 02:41:47,969 | test | INFO | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup started for test #1 test_volume_taf ========= 2021-01-30 02:42:29,045 | test | INFO | MainThread | [basetestcase:setUp:328] Initializing cluster 2021-01-30 02:42:33,911 | test | WARNING | MainThread | [basetestcase:_initialize_nodes:597] RAM quota was defined less than 100 MB: 2021-01-30 02:42:51,973 | test | INFO | MainThread | [basetestcase:setUp:350] Couchbase Cluster: default, Nodes: 172.23.105.175, 172.23.106.233, 172.23.106.236, 172.23.106.238, 172.23.106.250, 172.23.106.251, 172.23.121.74, 172.23.121.78 initialized 2021-01-30 02:42:52,000 | test | INFO | MainThread | [basetestcase:log_setup_status:552] ========= BaseTestCase setup finished for test #1 test_volume_taf ========= 2021-01-30 02:42:52,038 | test | INFO | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup started for test #1 test_volume_taf ========= 2021-01-30 02:43:00,213 | test | INFO | pool-1-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 172.23.105.175 | kv | Cluster node | | 172.23.106.233 | None | <--- IN --- | | 172.23.106.236 | None | <--- IN --- | | 172.23.106.238 | None | <--- IN --- | +----------------+----------+--------------+ 2021-01-30 02:43:00,234 | test | INFO | pool-1-thread-9 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-30 02:43:10,319 | test | INFO | pool-1-thread-9 | [task:check:322] Rebalance - status: none, progress: 100 2021-01-30 02:43:10,344 | test | INFO | pool-1-thread-9 | [task:check:379] Rebalance completed with progress: 100% in 10.131000042 sec 2021-01-30 02:43:10,615 | test | INFO | MainThread | [bucket_ready_functions:create_buckets_using_json_data:1778] Creating required buckets from template 2021-01-30 02:43:11,706 | test | INFO | MainThread | [bucket_ready_functions:wait_for_collection_creation_to_complete:2247] Waiting for all collections to be created 2021-01-30 02:43:11,946 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ | bucket1 | couchbase | 1 | none | 0 | 0 | 0 | 0 | 0 | | bucket2 | couchbase | 2 | none | 0 | 0 | 1258291200 | 0 | 0 | | bucket3 | ephemeral | 1 | none | 0 | 0 | 0 | 0 | 0 | | default | couchbase | 1 | none | 0 | 0 | 0 | 0 | 0 | +---------+-----------+----------+------------+-----+-------+------------+----------+-----------+ 2021-01-30 02:43:11,948 | test | INFO | MainThread | [collections_base:collection_setup:143] Creating required SDK clients for client_pool 2021-01-30 02:43:39,115 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-01-30 02:43:39,155 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-01-30 03:11:32,980 | test | INFO | pool-1-thread-15 | [table_view:display:72] Ops trend for bucket 'bucket1' +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******.................................................*****X | 10.000 | | 0.000 | ******************************************************************************.................................................*****X | 9.997 | | 0.000 | ***************.................................................*****X | 10.006 | | 0.000 | ************************************.................................................****X | 9.997 | | 0.000 | ***********.................................................****X | 10.000 | | 0.000 | *************************************************************************************************************************.................................................****X | 9.998 | | 0.000 | *****.................................................****X | 10.005 | | 7.405 | ...................................X | 7.405 | | 0.000 | ***************************************************************************************************************************.................................................*****.............*.....................................***X | 20.001 | | 5.200 | ........................X | 5.200 | | 0.000 | *********************************************************************************.................................................****X | 9.997 | | 2.600 | ...........X | 2.600 | | 0.000 | ****************************************************************************************************************************************************************************************************************....................................................................................................*****X | 20.014 | | 0.000 | ********************************************************************************************.................................................****X | 10.000 | | 2.600 | ...........X | 2.600 | | 0.000 | **********....................................*.............****X | 10.000 | | 0.000 | **************************************X | 0.000 | +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-01-30 03:11:33,994 | test | INFO | pool-1-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket2' +-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | *******....................................................................................................****X | 20.000 | | 5.200 | .........................X | 5.200 | | 0.000 | ***************************************************************************************************************************************************************************************************************************..................................................****X | 10.002 | | 0.000 | *************************************************************************************************************************************************************************************************************************************.................................................****X | 9.996 | | 0.000 | *****************.................................................****X | 9.995 | | 0.000 | ***********.................................................****X | 9.997 | | 2.600 | ............X | 2.600 | | 0.000 | **********************************************************************************************************************************************************..................................................****X | 10.000 | | 0.000 | **************************************************************************************************.................................................******.****X | 10.004 | | 9.993 | ................................................******.****X | 10.007 | | 0.000 | *************************************.........................................................................*..........................****X | 19.996 | | 0.000 | *****..................................................****X | 10.000 | | 7.400 | ....................................X | 7.400 | | 0.000 | ****************..................................................****X | 10.000 | | 0.000 | *********************************X | 0.000 | +-------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-01-30 03:11:34,775 | test | INFO | pool-1-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket3' +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | Min | Trend | Max | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ | 0.000 | **********************************************************************************************************************************...................................................................................................*****X | 29.999 | | 0.000 | ***********************************************************************************************************************************************.................................****X | 9.997 | | 2.600 | .......X | 2.600 | | 0.000 | ***************........*.........................****X | 9.998 | | 0.000 | ***************************.................................****X | 9.999 | | 0.000 | **********************************************************************************************************************************************************.................................****X | 10.002 | | 0.000 | ****************........................*.........**********X | 10.003 | | 0.000 | ******************************************************************************************************************************************.................................*****X | 10.004 | | 0.000 | **********************************************************************************.................................*****X | 10.007 | | 0.000 | ****************.................................****X | 10.000 | | 0.000 | ****************************....................................................................................................****X | 30.011 | | 22.211 | .........................................................................X | 22.211 | | 0.000 | ***************************************************************************************************X | 0.000 | +--------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+ 2021-01-30 03:11:35,887 | test | INFO | pool-1-thread-18 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | *******.................*.....................................................*****......................*****X | 50192.400 | | 47109.100 | .....................................................................................****....*..*****.****X | 50745.500 | | 49935.200 | ..........................................................................................*****.*****.****X | 50745.500 | | 50103.300 | ...........................................................................................******.***X | 50841.207 | | 50293.695 | ...........................................................................................******.****X | 51087.100 | | 47838.774 | ......................................................................................*****..****X | 48576.673 | | 48549.800 | ........................................................................................******...*****...******...***X | 53408.568 | | 52782.449 | ...............................................................................................*****X | 52782.449 | | 51026.748 | ............................................................................................****X | 51026.748 | | 48600.529 | ........................................................................................*****..*****...**********....*.****X | 54106.598 | | 53484.837 | .................................................................................................*****.*********X | 54363.449 | | 54013.900 | ..................................................................................................****X | 54013.900 | | 51897.276 | ..............................................................................................****X | 51897.276 | | 51164.729 | .............................................................................................*****X | 51164.729 | | 51019.024 | ............................................................................................*****.****X | 51243.805 | | 50757.366 | ............................................................................................*****..*****.**********X | 52727.057 | | 50224.104 | ...........................................................................................****X | 50224.104 | | 49317.393 | .........................................................................................****X | 49317.393 | | 44686.538 | .................................................................................******................****X | 53860.100 | | 50600.500 | ...........................................................................................*****..****X | 51219.577 | | 50610.287 | ...........................................................................................*****X | 50610.287 | | 49004.954 | .........................................................................................*****.........****X | 53884.134 | | 52532.241 | ...............................................................................................****X | 52532.241 | | 49867.476 | ..........................................................................................*****.*****.****X | 50921.073 | | 50017.300 | ..........................................................................................*****......*****..****X | 53895.967 | | 53522.867 | .................................................................................................X | 53522.867 | | 52372.620 | ...............................................................................................****X | 52372.620 | | 51476.258 | .............................................................................................***X | 51476.258 | | 51318.958 | .............................................................................................X | 51318.958 | | 50885.336 | ............................................................................................****X | 50885.336 | | 50584.043 | ...........................................................................................****X | 50584.043 | | 49503.627 | .........................................................................................*****.........****X | 54400.267 | | 53128.717 | ................................................................................................*****.****X | 53795.686 | | 52831.662 | ................................................................................................****X | 52831.662 | | 51306.428 | .............................................................................................X | 51306.428 | | 50815.451 | ............................................................................................*********X | 50994.080 | | 49941.416 | ..........................................................................................*****....*****X | 51739.700 | | 51519.818 | .............................................................................................***X | 51519.818 | | 51355.699 | .............................................................................................X | 51355.699 | | 51307.599 | .............................................................................................**********.****X | 52038.981 | | 51223.781 | .............................................................................................X | 51223.781 | | 48772.044 | ........................................................................................****X | 48772.044 | | 48749.061 | ........................................................................................*****..****X | 49997.650 | | 49500.337 | .........................................................................................*****X | 49500.337 | | 49108.900 | .........................................................................................*****.*..***X | 50873.800 | | 46041.278 | ...................................................................................X | 46041.278 | | 44416.278 | ................................................................................*****........******..*****.****X | 50562.600 | | 50327.109 | ...........................................................................................****X | 50327.109 | | 50034.899 | ..........................................................................................X | 50034.899 | | 49948.599 | ..........................................................................................*****..****X | 50749.903 | | 50461.303 | ...........................................................................................X | 50461.303 | | 49597.985 | ..........................................................................................****.*****X | 50586.061 | | 49230.061 | .........................................................................................**********X | 49402.282 | | 49222.156 | .........................................................................................******..****X | 50470.335 | | 49160.427 | .........................................................................................****X | 49160.427 | | 49000.253 | .........................................................................................****X | 49000.253 | | 46288.618 | ....................................................................................************.****..******.*****..****X | 49564.100 | | 48469.300 | ........................................................................................****X | 48469.300 | | 47685.709 | ......................................................................................****X | 47685.709 | | 47010.660 | .....................................................................................X | 47010.660 | | 46812.160 | .....................................................................................****X | 46812.160 | | 45243.974 | ..................................................................................*****.*..****.******...****X | 49466.220 | | 46895.700 | .....................................................................................****X | 46895.700 | | 42779.575 | .............................................................................*****X | 42779.575 | | 36439.699 | .................................................................****X | 36439.699 | | 33029.900 | ...........................................................X | 33029.900 | | 31905.300 | .........................................................****X | 31905.300 | | 28018.500 | ..................................................*****.*****X | 28585.478 | | 28512.068 | ...................................................****X | 28512.068 | | 27126.903 | ................................................X | 27126.903 | | 26659.432 | ...............................................*****....*****.*****X | 29344.932 | | 27259.450 | .................................................**********X | 27312.251 | | 27213.040 | ................................................****X | 27213.040 | | 26969.600 | ................................................X | 26969.600 | | 26192.300 | ...............................................****.******..*****.................******..................*****..............*****X | 54426.600 | | 53631.400 | .................................................................................................*********X | 53687.000 | | 52321.500 | ...............................................................................................******...****X | 53953.300 | | 52303.416 | ...............................................................................................****X | 52303.416 | | 49904.075 | ..........................................................................................*****X | 49904.075 | | 48603.649 | ........................................................................................*****.****.*****X | 49557.205 | | 45063.286 | .................................................................................**********...........*****X | 50661.564 | | 49621.903 | ..........................................................................................****X | 49621.903 | | 48923.229 | ........................................................................................*****X | 48923.229 | | 46988.967 | .....................................................................................*****...*****.*****.*.*****..****X | 51368.000 | | 50215.667 | ...........................................................................................*****..******.****X | 52247.326 | | 48938.682 | ........................................................................................****X | 48938.682 | | 48330.253 | .......................................................................................X | 48330.253 | | 48122.853 | .......................................................................................*****.....****X | 50711.520 | | 49926.500 | ..........................................................................................*****.*...****X | 52036.595 | | 51892.990 | ..............................................................................................****X | 51892.990 | | 51020.259 | ............................................................................................****X | 51020.259 | | 50903.000 | ............................................................................................****X | 50903.000 | | 50787.481 | ............................................................................................X | 50787.481 | | 50759.781 | ............................................................................................****X | 50759.781 | | 48861.992 | ........................................................................................****X | 48861.992 | | 46137.800 | ...................................................................................****X | 46137.800 | | 44704.725 | .................................................................................****X | 44704.725 | | 43190.692 | ..............................................................................X | 43190.692 | | 42680.668 | .............................................................................*****.........*****.*.****X | 48936.000 | | 47589.200 | ......................................................................................****X | 47589.200 | | 43437.900 | ..............................................................................X | 43437.900 | | 30967.300 | .......................................................*****.....*****....*****........*...*****....................*****..*X | 53639.000 | +-----------+-------------------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-01-30 03:11:37,473 | test | INFO | MainThread | [table_view:display:72] Cluster statistics +----------------+----------+-----------------+-------------+-------------+----------------+---------------------+-----------------------+ | Node | Services | CPU_utilization | Mem_total | Mem_free | Swap_mem_used | Active / Replica | Version | +----------------+----------+-----------------+-------------+-------------+----------------+---------------------+-----------------------+ | 172.23.105.175 | kv | 70.5502896261 | 25084055552 | 15369609216 | 0 / 3758092288 | 19974632 / 19952239 | 7.0.0-4342-enterprise | | 172.23.106.236 | kv | 75.6668410042 | 25084067840 | 15467679744 | 0 / 3758092288 | 19969905 / 19991624 | 7.0.0-4342-enterprise | | 172.23.106.233 | kv | 75.3367333595 | 25084071936 | 15465844736 | 0 / 3758092288 | 19970016 / 19979505 | 7.0.0-4342-enterprise | | 172.23.106.238 | kv | 75.0658587987 | 25084071936 | 15466074112 | 0 / 3758092288 | 19971475 / 19962971 | 7.0.0-4342-enterprise | +----------------+----------+-----------------+-------------+-------------+----------------+---------------------+-----------------------+ 2021-01-30 03:11:39,394 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:41,592 | test | INFO | pool-1-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:43,706 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:44,980 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:45,798 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:47,233 | test | INFO | pool-1-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:49,335 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:50,569 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:51,398 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:52,895 | test | INFO | pool-1-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:54,953 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:56,141 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:57,002 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:11:58,535 | test | INFO | pool-1-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:00,563 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:02,026 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:02,585 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:04,180 | test | INFO | pool-1-thread-23 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:06,187 | test | INFO | pool-1-thread-8 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:08,177 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:12:13,181 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4381] Validating collection stats and item counts 2021-01-30 03:12:27,476 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 0, Actual: 1 2021-01-30 03:12:27,476 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:12:35,220 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 0, Actual: 1 2021-01-30 03:12:35,220 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:12:42,782 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 0, Actual: 1 2021-01-30 03:12:42,782 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:12:50,477 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 0, Actual: 1 2021-01-30 03:12:50,479 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:16:07,197 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | bucket1 | couchbase | 1 | none | 0 | 1500 | 419430400 | 80830928 | 49849260 | | bucket2 | couchbase | 2 | none | 0 | 1500 | 1258291200 | 110552656 | 82500099 | | bucket3 | ephemeral | 1 | none | 0 | 1500 | 419430400 | 58131072 | 136 | | default | couchbase | 1 | none | 0 | 80000000 | 71303168000 | 31412965904 | 33709561612 | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ 2021-01-30 03:16:07,198 | test | INFO | MainThread | [basetestcase:log_setup_status:552] ========= CollectionBase setup complete for test #1 test_volume_taf ========= 2021-01-30 03:16:12,377 | test | INFO | MainThread | [Collections:test_volume_taf:381] Finished steps 1-4 successfully in setup 2021-01-30 03:16:12,378 | test | INFO | MainThread | [Collections:test_volume_taf:387] Step 5: Rebalance in with Loading of docs 2021-01-30 03:16:12,473 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-01-30 03:16:15,604 | test | INFO | pool-1-thread-6 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 172.23.105.175 | kv | Cluster node | | 172.23.106.236 | kv | Cluster node | | 172.23.106.233 | kv | Cluster node | | 172.23.106.238 | kv | Cluster node | | 172.23.121.78 | None | <--- IN --- | +----------------+----------+--------------+ 2021-01-30 03:16:15,632 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-30 03:16:20,707 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-30 03:16:25,611 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-01-30 03:16:25,778 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.0653594771242 2021-01-30 03:16:30,904 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.31862745098 2021-01-30 03:16:36,144 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:16:41,368 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:16:46,523 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:16:51,707 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:16:56,855 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:02,013 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:07,105 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:12,214 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:17,312 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:22,398 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:27,461 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:32,536 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:37,607 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:42,723 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.383986928105 2021-01-30 03:17:47,832 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.449346405229 2021-01-30 03:17:52,946 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.580065359477 2021-01-30 03:17:58,065 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.580065359477 2021-01-30 03:18:00,604 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in default:7G_VjXodJa1YkY9DUtoAS3tYNCOTy9sIws3XsAvfbgluXPpPlk7eq0%%6nE5KS-23-739000:Ta0J2YM2u2PdJCCZhRoH2q8eGCauDcT5zU%1iUQpX5MWRXsKKiiN40TXvt6cN4HP5ztIY0mqVhD3TUj%qZ7tFO869M6GfPjb7WXtsp-NMzXh1NSoEmE0vf2LlRmT-23-994000 +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | Read Key | Exception | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-161 | {'error': 'com.couchbase.client.core.error.DocumentNotFoundException | Document with the given id not found', 'cas': 0, 'value': {}} | +----------------------+--------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-30 03:18:00,605 | test | INFO | pool-1-thread-13 | [table_view:display:72] Keys failed in default:7G_VjXodJa1YkY9DUtoAS3tYNCOTy9sIws3XsAvfbgluXPpPlk7eq0%%6nE5KS-23-739000:Ta0J2YM2u2PdJCCZhRoH2q8eGCauDcT5zU%1iUQpX5MWRXsKKiiN40TXvt6cN4HP5ztIY0mqVhD3TUj%qZ7tFO869M6GfPjb7WXtsp-NMzXh1NSoEmE0vf2LlRmT-23-994000 +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Create Key | Exception | +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | test_collections-161 | com.couchbase.client.core.error.AmbiguousTimeoutException | InsertRequest, Reason: TIMEOUT | 172.23.121.78:11210 | reason:TIMEOUT | retryAttempts:1 | retryReasons:[COLLECTION_MAP_REFRESH_IN_PROGRESS] | +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 2021-01-30 03:18:03,194 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 0.694444444444 2021-01-30 03:18:05,677 | test | ERROR | pool-1-thread-15 | [task:call:3540] Failed to load 1 docs from 0 to 5000 2021-01-30 03:18:08,315 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 1.00490196078 2021-01-30 03:18:13,463 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 1.57679738562 2021-01-30 03:18:18,569 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 1.88725490196 2021-01-30 03:18:23,678 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 2.27124183007 2021-01-30 03:18:28,796 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 2.82679738562 2021-01-30 03:18:33,926 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 3.64379084967 2021-01-30 03:18:39,084 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 3.76633986928 2021-01-30 03:18:44,168 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 4.27287581699 2021-01-30 03:18:49,272 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 4.9591503268 2021-01-30 03:18:54,397 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 5.21241830065 2021-01-30 03:18:59,484 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 5.65359477124 2021-01-30 03:19:04,581 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 6.15196078431 2021-01-30 03:19:09,670 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 6.53594771242 2021-01-30 03:19:14,767 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 6.96895424837 2021-01-30 03:19:19,878 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 7.58986928105 2021-01-30 03:19:24,982 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 7.97385620915 2021-01-30 03:19:30,131 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 8.47222222222 2021-01-30 03:19:35,354 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 8.7908496732 2021-01-30 03:19:40,483 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 8.92156862745 2021-01-30 03:19:45,621 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 8.97875816993 2021-01-30 03:19:50,778 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 9.55882352941 2021-01-30 03:19:55,948 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 9.67320261438 2021-01-30 03:20:01,065 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 10.0490196078 2021-01-30 03:20:06,197 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 10.4901960784 2021-01-30 03:20:11,298 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 11.0539215686 2021-01-30 03:20:16,394 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 11.4379084967 2021-01-30 03:20:21,519 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 11.8790849673 2021-01-30 03:20:26,618 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 12.1895424837 2021-01-30 03:20:31,723 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 12.7614379085 2021-01-30 03:20:36,867 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 13.1290849673 2021-01-30 03:20:41,976 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 13.6683006536 2021-01-30 03:20:47,046 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 13.8562091503 2021-01-30 03:20:52,122 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 14.3545751634 2021-01-30 03:20:57,220 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 14.6568627451 2021-01-30 03:21:02,368 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 15.0245098039 2021-01-30 03:21:07,509 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 15.6454248366 2021-01-30 03:21:12,664 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 16.0130718954 2021-01-30 03:21:17,727 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 16.1928104575 2021-01-30 03:21:22,828 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 16.7973856209 2021-01-30 03:21:27,927 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 16.9199346405 2021-01-30 03:21:33,023 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 17.3366013072 2021-01-30 03:21:38,121 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 17.818627451 2021-01-30 03:21:43,206 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 18.2352941176 2021-01-30 03:21:48,344 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 18.5294117647 2021-01-30 03:21:53,467 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 19.0196078431 2021-01-30 03:21:58,575 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 19.1339869281 2021-01-30 03:22:03,727 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 19.4852941176 2021-01-30 03:22:08,861 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 20.0245098039 2021-01-30 03:22:13,953 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 20.2614379085 2021-01-30 03:22:19,049 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 20.8496732026 2021-01-30 03:22:24,112 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 20.9722222222 2021-01-30 03:22:29,232 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 21.3888888889 2021-01-30 03:22:34,313 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 21.7401960784 2021-01-30 03:22:39,450 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 22.2875816993 2021-01-30 03:22:44,561 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 22.3529411765 2021-01-30 03:22:49,634 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 22.6552287582 2021-01-30 03:22:54,719 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 23.0637254902 2021-01-30 03:22:59,845 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 23.3578431373 2021-01-30 03:23:04,960 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 23.4232026144 2021-01-30 03:23:10,061 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 24.0277777778 2021-01-30 03:23:15,217 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 24.3137254902 2021-01-30 03:23:20,312 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 24.3137254902 2021-01-30 03:23:25,453 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 24.3137254902 2021-01-30 03:23:30,561 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 24.3137254902 2021-01-30 03:23:35,706 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 25.0 2021-01-30 03:23:40,855 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 34.4444444444 2021-01-30 03:23:45,967 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 43.7581699346 2021-01-30 03:23:51,118 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 50.2532679739 2021-01-30 03:23:56,285 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 59.4444444444 2021-01-30 03:24:01,395 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 68.5049019608 2021-01-30 03:24:06,892 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 75.0 2021-01-30 03:24:12,069 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 80.428377966 2021-01-30 03:24:17,226 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 85.6709391928 2021-01-30 03:24:22,395 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 90.4213015946 2021-01-30 03:24:27,525 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 95.3993095293 2021-01-30 03:24:32,990 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: running, progress: 100 2021-01-30 03:24:43,191 | test | INFO | pool-1-thread-6 | [task:check:322] Rebalance - status: none, progress: 100 2021-01-30 03:24:43,286 | test | INFO | pool-1-thread-6 | [task:check:379] Rebalance completed with progress: 100% in 507.682999849 sec 2021-01-30 03:35:09,767 | test | INFO | pool-1-thread-16 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+------------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***********X | 9.500 | | 2.500 | X | 2.500 | | 0.000 | *****....................................................................................................****X | 1007.449 | | 2.599 | **********X | 2.901 | | 1.900 | *****.................................................*****X | 500.000 | | 3.997 | *****..................................................*****X | 509.708 | | 0.000 | *****..................................................****X | 503.900 | | 126.200 | ...........X | 126.200 | | 0.000 | ***************X | 3.598 | | 0.000 | *********************X | 3.800 | | 2.000 | *****X | 2.000 | | 0.000 | *********X | 7.001 | | 1.900 | ****X | 1.900 | | 0.500 | X | 0.500 | | 0.000 | **********X | 2.001 | | 0.000 | **********************************************************************.................................................****X | 499.365 | | 0.000 | *****************************X | 2.001 | | 0.400 | X | 0.400 | | 0.000 | ****************************************************************************************X | 1.900 | | 1.900 | X | 1.900 | | 1.899 | ****X | 1.899 | | 1.601 | ****X | 1.601 | | 0.000 | ***************************X | 3.598 | | 0.000 | ************************************X | 2.000 | | 1.700 | X | 1.700 | | 0.000 | **************************************************************************X | 2.000 | | 0.000 | **********X | 2.001 | | 0.000 | **********X | 1.599 | | 0.000 | ********************X | 4.798 | | 0.000 | ********************X | 1.600 | | 0.000 | *************************************X | 1.600 | | 0.000 | *************X | 0.000 | +---------+------------------------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:35:09,808 | test | INFO | pool-1-thread-17 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | **.*****...................................................................................................****X | 1007.400 | | 756.400 | ..........................................................................X | 756.400 | | 3.900 | ****.*****X | 14.998 | | 1.899 | ******..................................................****X | 505.739 | | 0.000 | *********************..........................................................................*.........................****X | 1005.800 | | 500.000 | ................................................****X | 500.000 | | 0.000 | *************************************X | 1.600 | | 0.000 | *********************************************************************X | 2.000 | | 1.400 | X | 1.400 | | 0.000 | ***********************************************************************X | 3.499 | | 2.699 | X | 2.699 | | 0.000 | *********************************************************.......................................*..........****X | 500.284 | | 0.000 | ***************X | 1.600 | | 0.000 | ****************************************************************************X | 5.199 | | 0.000 | ****************************************************************************************************************************************************************************************************X | 0.000 | +---------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:35:10,795 | test | INFO | pool-1-thread-18 | [table_view:display:72] Ops trend for bucket 'bucket3' +---------+---------------------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ******X | 1.900 | | 0.000 | ***********....................................................................................................****X | 1013.349 | | 253.900 | ........................X | 253.900 | | 0.000 | *****.................................................*****X | 503.939 | | 0.000 | ****************.........................*..........................................................................****X | 1008.207 | | 0.000 | ***************X | 1.899 | | 0.000 | ****************X | 1.901 | | 0.000 | *******X | 7.404 | | 7.404 | ***.................................................****X | 501.875 | | 500.025 | ................................................*****X | 500.025 | | 0.000 | *******************************************X | 1.900 | | 0.000 | **********X | 5.497 | | 2.001 | ****X | 2.001 | | 0.000 | ***************X | 3.599 | | 3.503 | *****X | 3.503 | | 0.000 | ***************X | 1.900 | | 1.400 | X | 1.400 | | 0.000 | ********************X | 2.000 | | 0.600 | X | 0.600 | | 0.000 | *************X | 1.901 | | 0.000 | ********************************************************************X | 1.901 | | 0.000 | ********************************X | 1.901 | | 0.000 | ***************************X | 5.497 | | 0.000 | *********************X | 1.901 | | 0.000 | ***************X | 1.600 | | 1.400 | X | 1.400 | | 0.000 | ********************X | 1.600 | | 0.000 | ********************************X | 2.000 | | 0.000 | *****************************************************X | 5.200 | | 0.000 | *************************************X | 3.600 | | 0.000 | **************************X | 3.600 | | 0.000 | ********************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:35:11,108 | test | INFO | pool-1-thread-14 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | **........*****.............................................................................*****.****X | 56485.500 | | 50725.419 | ............................................................................****X | 50725.419 | | 50127.332 | ...........................................................................X | 50127.332 | | 49919.632 | ...........................................................................****X | 49919.632 | | 41455.507 | ..............................................................****X | 41455.507 | | 37103.817 | .......................................................X | 37103.817 | | 37103.817 | .......................................................*X | 37103.817 | | 37103.817 | .......................................................**...*.......****......*..*****.............****X | 57057.743 | | 33193.800 | .................................................******..........*****.....*****X | 42817.180 | | 38099.313 | .........................................................*****............**********X | 46204.505 | | 43494.421 | .................................................................****X | 43494.421 | | 43134.627 | .................................................................*****.............****X | 51899.015 | | 47769.041 | ........................................................................*****X | 47769.041 | | 43604.798 | .................................................................*****......*****..****X | 48401.100 | | 45567.048 | ....................................................................*****X | 45567.048 | | 45082.496 | ....................................................................****X | 45082.496 | | 38509.907 | ..........................................................X | 38509.907 | | 38509.907 | ..........................................................*X | 38509.907 | | 38509.907 | ..........................................................*X | 38509.907 | | 38509.907 | ..........................................................*.........*****.....****X | 48040.500 | | 29141.500 | ...........................................******.....................*****...****X | 44816.355 | | 39806.200 | ............................................................******..........*****..***X | 47966.444 | | 47966.444 | ........................................................................X | 47966.444 | | 44456.544 | ...................................................................X | 44456.544 | | 32783.764 | .................................................*****..................***X | 44904.207 | | 44904.207 | ...................................................................*....*****X | 47040.592 | | 39371.622 | ...........................................................X | 39371.622 | | 39371.622 | ...........................................................****.......................**X | 54458.085 | | 54458.085 | ..................................................................................*X | 54458.085 | | 52796.385 | ...............................................................................X | 52796.385 | | 46923.841 | ......................................................................****X | 46923.841 | | 46761.900 | ......................................................................*****.............****X | 54981.200 | | 50946.509 | .............................................................................X | 50946.509 | | 49929.809 | ...........................................................................****.........*****.****X | 56227.732 | | 52747.869 | ...............................................................................*X | 52747.869 | | 52747.869 | ...............................................................................*X | 52747.869 | | 39243.278 | ...........................................................*****......****.*****..........*****..*****X | 52073.134 | | 49069.997 | ..........................................................................********X | 49119.134 | | 49119.134 | ..........................................................................*.*...*****.....****X | 54896.000 | | 47693.341 | ........................................................................*****..*****X | 49093.283 | | 45472.268 | ....................................................................*****.*****..*.*****X | 48248.248 | | 48248.248 | ........................................................................****.*****........*.****X | 54733.731 | | 49538.974 | ..........................................................................***X | 49538.974 | | 49538.974 | ..........................................................................X | 49538.974 | | 44604.900 | ...................................................................****X | 44604.900 | | 42629.200 | ................................................................*****X | 42629.200 | | 41998.600 | ...............................................................*****.*..***X | 44274.300 | | 44138.425 | ..................................................................******..........****X | 50439.426 | | 48538.900 | .........................................................................*****....*****X | 51071.066 | | 45017.682 | ....................................................................*****...*****.*...***X | 49953.605 | | 42124.905 | ...............................................................X | 42124.905 | | 40204.351 | ............................................................*****....*****.******.......*X | 48109.001 | | 48109.001 | ........................................................................***..*.....***X | 52385.635 | | 52103.763 | ..............................................................................******.....******.****X | 55467.123 | | 51784.224 | ..............................................................................*X | 51784.224 | | 51784.224 | ..............................................................................**X | 51784.224 | | 45358.265 | ....................................................................****X | 45358.265 | | 40536.997 | .............................................................*****X | 40536.997 | | 35807.124 | .....................................................*X | 35807.124 | | 35807.124 | .....................................................***........****X | 40624.600 | | 32920.764 | .................................................******.........X | 39013.520 | | 39013.520 | ..........................................................*X | 39013.520 | | 39013.520 | ..........................................................*X | 39013.520 | | 37651.880 | ........................................................*X | 37651.880 | | 37651.880 | ........................................................*X | 37651.880 | | 37651.880 | ........................................................*...............................*****X | 57942.819 | | 57684.200 | .......................................................................................*********X | 57747.411 | | 57679.699 | .......................................................................................X | 57679.699 | | 57647.405 | .......................................................................................*****......**X | 61555.600 | | 61555.600 | .............................................................................................**......****X | 65237.991 | | 53622.312 | .................................................................................X | 53622.312 | | 53622.312 | .................................................................................*X | 53622.312 | | 53622.312 | .................................................................................*X | 53622.312 | | 53622.312 | .................................................................................*........****X | 59067.091 | | 56305.190 | .....................................................................................*****....*****X | 58747.206 | | 58078.948 | ........................................................................................****X | 58078.948 | | 56145.688 | .....................................................................................*****.....*****.*****X | 60384.400 | | 54799.528 | ..................................................................................*****....*****...*****X | 59262.520 | | 55113.708 | ...................................................................................*****..****X | 56403.090 | | 55811.200 | ....................................................................................****X | 55811.200 | | 55182.600 | ...................................................................................X | 55182.600 | | 52648.155 | ...............................................................................***X | 52648.155 | +-----------+-------------------------------------------------------------------------------------------------------------+-----------+ 2021-01-30 03:35:12,940 | test | WARNING | Thread-3489 | [bucket_ready_functions:validate_crud_task_per_collection:500] default:7G_VjXodJa1YkY9DUtoAS3tYNCOTy9sIws3XsAvfbgluXPpPlk7eq0%%6nE5KS-23-739000:Ta0J2YM2u2PdJCCZhRoH2q8eGCauDcT5zU%1iUQpX5MWRXsKKiiN40TXvt6cN4HP5ztIY0mqVhD3TUj%qZ7tFO869M6GfPjb7WXtsp-NMzXh1NSoEmE0vf2LlRmT-23-994000 create failed keys from task: [u'test_collections-161'] 2021-01-30 03:35:18,016 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:20,872 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:23,403 | test | INFO | pool-1-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:23,619 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:25,894 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:26,414 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:28,049 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:29,009 | test | INFO | pool-1-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:30,733 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:31,496 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:31,936 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:33,743 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:34,549 | test | INFO | pool-1-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:36,293 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:37,048 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:37,479 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:39,285 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:40,141 | test | INFO | pool-1-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:41,795 | test | INFO | pool-1-thread-7 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:42,621 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:43,062 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:44,882 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:45,696 | test | INFO | pool-1-thread-6 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:48,187 | test | INFO | pool-1-thread-19 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:50,444 | test | INFO | pool-1-thread-28 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:35:55,446 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4381] Validating collection stats and item counts 2021-01-30 03:36:12,441 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 12, Actual: d 2021-01-30 03:36:12,444 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:36:22,265 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 11, Actual: c 2021-01-30 03:36:22,266 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:36:31,921 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 11, Actual: c 2021-01-30 03:36:31,921 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:36:41,562 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 56, Actual: 39 2021-01-30 03:36:41,562 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:40:09,203 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | bucket1 | couchbase | 1 | none | 0 | 31248 | 524288000 | 92710024 | 123418968 | | bucket2 | couchbase | 2 | none | 0 | 35864 | 1572864000 | 126785144 | 168975030 | | bucket3 | ephemeral | 1 | none | 0 | 36344 | 524288000 | 88363208 | 170 | | default | couchbase | 1 | none | 0 | 69841200 | 89128960000 | 27835300592 | 29386587987 | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ 2021-01-30 03:40:09,204 | test | INFO | MainThread | [Collections:test_volume_taf:395] Step 6: Rebalance Out with Loading of docs 2021-01-30 03:40:09,345 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-01-30 03:40:09,403 | test | INFO | pool-1-thread-14 | [table_view:display:72] Rebalance Overview +----------------+----------+--------------+ | Nodes | Services | Status | +----------------+----------+--------------+ | 172.23.105.175 | kv | Cluster node | | 172.23.121.78 | kv | Cluster node | | 172.23.106.236 | kv | Cluster node | | 172.23.106.233 | kv | Cluster node | | 172.23.106.238 | [u'kv'] | --- OUT ---> | +----------------+----------+--------------+ 2021-01-30 03:40:09,433 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0 2021-01-30 03:40:14,516 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.309581778018 2021-01-30 03:40:19,581 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.627028638506 2021-01-30 03:40:23,035 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations 2021-01-30 03:40:24,657 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:29,726 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:34,924 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:40,076 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:45,187 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:50,361 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:40:55,500 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:41:00,657 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:41:05,805 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:41:10,954 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.888466547003 2021-01-30 03:41:16,084 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 0.953826024127 2021-01-30 03:41:21,207 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 1.08454497838 2021-01-30 03:41:26,293 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 1.28062340975 2021-01-30 03:41:31,374 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 1.411342364 2021-01-30 03:41:36,438 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 1.72050246797 2021-01-30 03:41:41,559 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 2.16059099414 2021-01-30 03:41:46,657 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 2.65690615468 2021-01-30 03:41:51,737 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 3.380657897 2021-01-30 03:41:56,802 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 4.12204418281 2021-01-30 03:42:01,891 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 4.66692762507 2021-01-30 03:42:06,964 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 5.41576005745 2021-01-30 03:42:12,144 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 5.96169905389 2021-01-30 03:42:17,211 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 6.64559094505 2021-01-30 03:42:22,280 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 7.50069278361 2021-01-30 03:42:27,375 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 8.2318851962 2021-01-30 03:42:32,469 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 8.9807121523 2021-01-30 03:42:37,548 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 9.47702731284 2021-01-30 03:42:42,628 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 10.0623026742 2021-01-30 03:42:47,707 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 10.3228972346 2021-01-30 03:42:52,822 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 10.7146296852 2021-01-30 03:42:57,953 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 11.1059431998 2021-01-30 03:43:03,030 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 11.4951456061 2021-01-30 03:43:08,111 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 11.9991191193 2021-01-30 03:43:13,200 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 12.3233864607 2021-01-30 03:43:18,299 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 12.647653802 2021-01-30 03:43:23,365 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 12.7766860601 2021-01-30 03:43:28,471 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 12.9702344472 2021-01-30 03:43:33,563 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 13.0992667052 2021-01-30 03:43:38,657 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 13.2928150923 2021-01-30 03:43:43,769 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 13.4218473504 2021-01-30 03:43:48,869 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 13.8681197391 2021-01-30 03:43:53,947 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 14.565211751 2021-01-30 03:43:59,066 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 15.159506823 2021-01-30 03:44:04,140 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 15.7126742836 2021-01-30 03:44:09,213 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 16.3801939153 2021-01-30 03:44:14,312 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 17.1678279065 2021-01-30 03:44:19,405 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 17.8992325252 2021-01-30 03:44:24,526 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 18.5003426018 2021-01-30 03:44:29,611 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 19.0944254678 2021-01-30 03:44:34,698 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 19.8673711575 2021-01-30 03:44:39,799 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 20.224675131 2021-01-30 03:44:44,875 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 20.6799699686 2021-01-30 03:44:49,967 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 21.0219681893 2021-01-30 03:44:55,045 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 21.363123062 2021-01-30 03:45:00,157 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 21.8184178995 2021-01-30 03:45:05,250 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 22.1595727722 2021-01-30 03:45:10,349 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 22.4436576624 2021-01-30 03:45:15,460 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 22.784812535 2021-01-30 03:45:20,532 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 23.0684757512 2021-01-30 03:45:25,602 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 23.5241922628 2021-01-30 03:45:30,691 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 23.8078554789 2021-01-30 03:45:35,769 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 24.2631503165 2021-01-30 03:45:40,891 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 24.2631503165 2021-01-30 03:45:46,015 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 24.3197986249 2021-01-30 03:45:51,134 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 24.3197986249 2021-01-30 03:45:56,247 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 24.9433516916 2021-01-30 03:46:01,346 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 25.0 2021-01-30 03:46:06,480 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 25.0 2021-01-30 03:46:11,609 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 40.961363264 2021-01-30 03:46:16,743 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 50.0 2021-01-30 03:46:21,855 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 50.0 2021-01-30 03:46:26,957 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 52.9730211757 2021-01-30 03:46:32,071 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 69.2401502145 2021-01-30 03:46:37,173 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 75.0 2021-01-30 03:46:42,282 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 75.0 2021-01-30 03:46:47,398 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 76.2433213285 2021-01-30 03:46:52,519 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 81.4378925244 2021-01-30 03:46:57,605 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 86.7587232729 2021-01-30 03:47:02,714 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 91.2071978984 2021-01-30 03:47:07,816 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 95.6961813527 2021-01-30 03:47:12,976 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 99.1796761749 2021-01-30 03:47:18,145 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 100 2021-01-30 03:47:23,266 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: running, progress: 100 2021-01-30 03:47:33,464 | test | INFO | pool-1-thread-14 | [task:check:322] Rebalance - status: none, progress: 100 2021-01-30 03:47:33,661 | test | INFO | pool-1-thread-14 | [task:check:379] Rebalance completed with progress: 100% in 444.257999897 sec 2021-01-30 03:55:36,857 | test | INFO | pool-1-thread-22 | [table_view:display:72] Ops trend for bucket 'bucket1' +---------+--------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+--------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***................................................................................*....................***X | 2071.558 | | 402.661 | ..................X | 402.661 | | 1.802 | *****.........................................................****X | 1191.248 | | 790.233 | .....................................*****X | 790.233 | | 8.901 | ****X | 8.901 | | 7.301 | X | 7.301 | | 0.000 | *********...................*.....****X | 500.067 | | 5.501 | *****....****X | 99.998 | | 80.098 | ..X | 80.098 | | 0.000 | ***************.............****X | 276.000 | | 0.000 | ********************************X | 1.500 | | 1.300 | X | 1.300 | | 0.000 | *****************************************************........................****X | 500.000 | | 79.984 | ..****X | 79.984 | | 0.000 | ******....****X | 96.053 | | 0.000 | *****************************X | 3.700 | | 2.700 | X | 2.700 | | 0.000 | ***********........****X | 179.929 | | 101.848 | ...****X | 101.848 | | 77.048 | ..X | 77.048 | | 1.799 | *********X | 3.301 | | 0.000 | *****************....****X | 97.834 | | 0.000 | *********************X | 3.301 | | 0.700 | X | 0.700 | | 0.000 | ****************....*****X | 83.700 | | 0.000 | ***************X | 2.299 | | 2.001 | *****X | 3.301 | | 1.900 | ****X | 1.900 | | 0.000 | ***************X | 3.702 | | 1.000 | X | 1.000 | | 0.000 | ***************X | 3.301 | | 0.000 | *****************....****X | 101.740 | | 1.500 | **********X | 1.901 | | 1.500 | *****X | 1.500 | | 0.000 | ****************X | 4.899 | | 0.000 | ******...****X | 81.480 | | 3.401 | ****X | 3.401 | | 2.700 | X | 2.700 | | 0.000 | *******************************X | 1.900 | | 0.000 | **********X | 1.500 | | 0.000 | *X | 0.000 | +---------+--------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:55:37,045 | test | INFO | pool-1-thread-10 | [table_view:display:72] Ops trend for bucket 'bucket2' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***...............................................................................*.....................****X | 2003.997 | | 292.204 | .............****X | 292.204 | | 230.167 | ..........X | 230.167 | | 0.000 | *****.....................****X | 425.543 | | 80.106 | ..*****.......*........................***X | 697.200 | | 150.398 | ......X | 150.398 | | 8.796 | ****X | 8.796 | | 0.000 | ***************X | 3.400 | | 0.000 | ******************************************X | 3.600 | | 0.000 | ***************X | 1.800 | | 0.000 | ***************************....*****....****X | 176.010 | | 0.000 | ***************X | 1.900 | | 1.900 | *****X | 1.900 | | 0.000 | ***************X | 1.800 | | 0.000 | **********X | 1.800 | | 0.000 | ************************X | 1.501 | | 0.000 | **********************....****X | 96.000 | | 1.900 | *****X | 1.900 | | 0.000 | ***************************************....*****..*...****X | 193.500 | | 0.000 | *********************.*....*****........****X | 261.900 | | 194.700 | ........X | 194.700 | | 0.000 | *******************************X | 1.500 | | 1.200 | X | 1.200 | | 0.000 | *********************..............*.....****X | 392.042 | | 291.842 | .............X | 291.842 | | 0.000 | **************************X | 1.500 | | 0.000 | ***********...****X | 80.021 | | 59.106 | .X | 59.106 | | 0.000 | **************************.*..****X | 80.000 | | 0.000 | ***********....*X | 100.000 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:55:37,322 | test | INFO | pool-1-thread-3 | [table_view:display:72] Ops trend for bucket 'bucket3' +---------+---------------------------------------------------------------------------------------------------------------+----------+ | Min | Trend | Max | +---------+---------------------------------------------------------------------------------------------------------------+----------+ | 0.000 | ***................................................................................*....................****X | 4407.534 | | 477.932 | .........****X | 477.932 | | 3.597 | *****X | 3.597 | | 0.000 | **********X | 3.600 | | 0.000 | ***************X | 1.900 | | 0.000 | *********X | 3.400 | | 0.600 | X | 0.600 | | 0.000 | ***************X | 3.702 | | 0.000 | ***************************************************************************************X | 1.499 | | 1.299 | X | 1.299 | | 0.000 | **************X | 1.800 | | 1.499 | *****X | 1.499 | | 0.000 | ***************X | 1.800 | | 0.000 | *******************X | 3.700 | | 0.000 | ****************X | 1.799 | | 0.000 | **********X | 1.500 | | 0.400 | **********X | 1.800 | | 0.000 | **********X | 1.499 | | 0.000 | **********X | 1.800 | | 1.800 | *****X | 1.801 | | 0.000 | **********X | 1.500 | | 0.000 | ***************************X | 1.900 | | 0.000 | *************************************************X | 1.901 | | 0.000 | ****************X | 3.699 | | 1.900 | ****X | 1.900 | | 0.500 | X | 0.500 | | 0.000 | ********************************X | 1.500 | | 1.500 | **********X | 1.500 | | 0.000 | **********************X | 1.500 | | 0.000 | ***********************************X | 0.000 | +---------+---------------------------------------------------------------------------------------------------------------+----------+ 2021-01-30 03:55:37,671 | test | INFO | pool-1-thread-21 | [table_view:display:72] Ops trend for bucket 'default' +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | Min | Trend | Max | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ | 0.000 | ***....*.....................****.....................................*..........*****......*****....*****......****X | 57067.414 | | 55374.814 | ....................................................................................X | 55374.814 | | 48379.300 | ..........................................................................***X | 48379.300 | | 45805.027 | ......................................................................**X | 45805.027 | | 45805.027 | ......................................................................**X | 45805.027 | | 35859.657 | ......................................................****X | 35859.657 | | 32697.317 | .................................................*****......****X | 36684.698 | | 36012.580 | ......................................................*****X | 36012.580 | | 33595.898 | ...................................................****X | 33595.898 | | 33164.998 | ..................................................X | 33164.998 | | 31314.799 | ...............................................****....*.*****...*****..*......***X | 41397.665 | | 39550.953 | ............................................................X | 39550.953 | | 39055.853 | ...........................................................*****........*****...******....X | 48432.900 | | 48432.900 | ..........................................................................**X | 48432.900 | | 48432.900 | ..........................................................................X | 48432.900 | | 41036.585 | ..............................................................*X | 41036.585 | | 41036.585 | ..............................................................**X | 41036.585 | | 27564.867 | .........................................*****......*..****X | 32636.129 | | 28490.900 | ...........................................*****.......*****...*****..................****X | 47003.259 | | 45912.016 | ......................................................................****X | 45912.016 | | 45218.744 | .....................................................................*****X | 45218.744 | | 43924.431 | ...................................................................*****.............*****.***X | 53433.958 | | 53433.958 | .................................................................................X | 53433.958 | | 46321.659 | ......................................................................X | 46321.659 | | 46321.659 | ......................................................................*X | 46321.659 | | 46321.659 | ......................................................................***..........*X | 52232.816 | | 52092.316 | ...............................................................................**X | 52092.316 | | 48686.460 | ..........................................................................*****.....*****...*****...****X | 55772.800 | | 49577.253 | ...........................................................................***X | 49577.253 | | 49323.653 | ...........................................................................X | 49323.653 | | 48539.300 | ..........................................................................****.........***X | 54553.708 | | 54553.708 | ...................................................................................*X | 54553.708 | | 54340.360 | ...................................................................................****X | 54340.360 | | 52560.433 | ................................................................................*****...*****...*****...*.*****.......****X | 63267.200 | | 61673.000 | ..............................................................................................*X | 61673.000 | | 61673.000 | ..............................................................................................***.*....****X | 64463.862 | | 60026.331 | ............................................................................................**X | 60026.331 | | 60026.331 | ............................................................................................**..*.********X | 62430.969 | | 62430.969 | ...............................................................................................*.*****X | 62923.035 | | 57218.830 | .......................................................................................****X | 57218.830 | | 52585.547 | ................................................................................X | 52585.547 | | 51105.047 | ..............................................................................*****........*****..*......****X | 61587.500 | | 60618.700 | .............................................................................................****X | 60618.700 | | 60008.600 | ............................................................................................*****.*.X | 61604.400 | | 61604.400 | ..............................................................................................***X | 61604.400 | | 61033.146 | .............................................................................................*****X | 61033.146 | | 60705.559 | .............................................................................................****X | 60705.559 | | 60623.400 | .............................................................................................X | 60623.400 | | 60262.300 | ............................................................................................*****.*X | 61112.577 | | 61112.577 | .............................................................................................**X | 61112.577 | | 58859.439 | ..........................................................................................*****X | 58859.439 | | 57673.954 | ........................................................................................*****.....X | 60900.635 | | 60900.635 | .............................................................................................**X | 60900.635 | | 60900.635 | .............................................................................................*X | 60900.635 | | 60086.388 | ............................................................................................****X | 60086.388 | | 57338.265 | .......................................................................................X | 57338.265 | | 56476.765 | ......................................................................................*****......****X | 60404.800 | | 59375.373 | ...........................................................................................*X | 59375.373 | | 59375.373 | ...........................................................................................****......****X | 63740.686 | | 58553.226 | .........................................................................................*****X | 58553.226 | | 57367.837 | .......................................................................................****X | 57367.837 | | 56006.156 | .....................................................................................X | 56006.156 | | 55487.556 | .....................................................................................****X | 55487.556 | | 55261.456 | ....................................................................................X | 55261.456 | | 54446.108 | ...................................................................................*****..****X | 55987.663 | | 55727.463 | .....................................................................................X | 55727.463 | | 54725.600 | ...................................................................................*****..........****X | 60623.595 | | 56995.179 | .......................................................................................****X | 56995.179 | | 56117.002 | ......................................................................................X | 56117.002 | | 55848.351 | .....................................................................................****X | 55848.351 | | 54176.434 | ...................................................................................*X | 54176.434 | | 54176.434 | ...................................................................................*X | 54176.434 | | 54176.434 | ...................................................................................X | 54176.434 | | 53957.800 | ..................................................................................******....****X | 56450.161 | | 50139.293 | ............................................................................*****X | 50139.293 | | 47347.400 | ........................................................................*****.......*..****X | 53226.500 | | 52075.200 | ...............................................................................****X | 52075.200 | | 51396.300 | ..............................................................................X | 51396.300 | | 51168.269 | ..............................................................................*****.......*****..**X | 57253.600 | +-----------+-----------------------------------------------------------------------------------------------------------------------------+-----------+ 2021-01-30 03:55:40,148 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:42,293 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:44,444 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:45,769 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:46,398 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:47,898 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:50,032 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:51,361 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:51,979 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:53,588 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:55,700 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:56,961 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:57,588 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:55:59,190 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:01,309 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:02,555 | test | INFO | pool-1-thread-17 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:03,187 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:04,792 | test | INFO | pool-1-thread-13 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:06,961 | test | INFO | pool-1-thread-30 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:08,943 | test | INFO | pool-1-thread-16 | [common_lib:sleep:22] Sleep 5 seconds. Reason: MC is down. Retrying.. 'num_items_for_persistence' 2021-01-30 03:56:13,944 | test | INFO | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4381] Validating collection stats and item counts 2021-01-30 03:56:28,012 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 22, Actual: 17 2021-01-30 03:56:28,013 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:56:35,805 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 23, Actual: 18 2021-01-30 03:56:35,805 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:56:44,035 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 22, Actual: 17 2021-01-30 03:56:44,036 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:56:52,242 | test | ERROR | MainThread | [bucket_ready_functions:validate_manifest_uid:4263] Bucket UID mismatch. Expected: 113, Actual: 72 2021-01-30 03:56:52,243 | test | WARNING | MainThread | [bucket_ready_functions:validate_docs_per_collections_all_buckets:4391] Bucket manifest UID mismatch! 2021-01-30 03:59:06,993 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | bucket1 | couchbase | 1 | none | 0 | 69996 | 419430400 | 101241344 | 228087840 | | bucket2 | couchbase | 2 | none | 0 | 59628 | 1258291200 | 136092032 | 231319742 | | bucket3 | ephemeral | 1 | none | 0 | 55104 | 419430400 | 100969808 | 136 | | default | couchbase | 1 | none | 0 | 60294576 | 71303168000 | 24280755816 | 26597605222 | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ 2021-01-30 03:59:06,994 | test | INFO | MainThread | [Collections:test_volume_taf:403] Step 7: Rebalance In_Out with Loading of docs 2021-01-30 03:59:07,059 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4624] Performing scope/collection specific operations 2021-01-30 03:59:13,539 | test | ERROR | pool-1-thread-14 | [rest_client:_http_request:747] POST http://172.23.105.175:8091/controller/addNode body: hostname=http%3A%2F%2F172.23.121.74%3A8091&password=password&user=Administrator headers: {'Accept': '*/*', 'Connection': 'close', 'Authorization': 'Basic QWRtaW5pc3RyYXRvcjpwYXNzd29yZA==\n', 'Content-Type': 'application/x-www-form-urlencoded'} error: 400 reason: unknown ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] auth: Administrator:password 2021-01-30 03:59:13,588 | test | INFO | pool-1-thread-14 | [rest_client:print_UI_logs:2593] Latest logs from UI on 172.23.105.175: 2021-01-30 03:59:13,588 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 5, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.105.175', u'tstamp': 1612007953524L, u'shortText': u'message', u'serverTime': u'2021-01-30T03:59:13.524Z', u'text': u'Failed to add node 172.23.121.74:8091 to cluster. Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: "[\\"Unexpected server error, request logged.\\"]"'} 2021-01-30 03:59:13,589 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_cluster', u'type': u'critical', u'node': u'ns_1@172.23.105.175', u'tstamp': 1612007953478L, u'shortText': u'message', u'serverTime': u'2021-01-30T03:59:13.478Z', u'text': u'Add transaction of \'ns_1@172.23.121.74\' failed because of {error,\n complete_join,\n <<"Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {error,rest_error,\n <<"Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \\"[\\\\\\"Unexpected server error, request logged.\\\\\\"]\\"">>,\n {bad_status,500,\n <<"[\\"Unexpected server error, request logged.\\"]">>}}}'} 2021-01-30 03:59:13,589 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.106.236', u'tstamp': 1612007948351L, u'shortText': u'node up', u'serverTime': u'2021-01-30T03:59:08.351Z', u'text': u"Node 'ns_1@172.23.106.236' saw that node 'ns_1@172.23.121.74' came up. Tags: []"} 2021-01-30 03:59:13,589 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.121.78', u'tstamp': 1612007948350L, u'shortText': u'node up', u'serverTime': u'2021-01-30T03:59:08.350Z', u'text': u"Node 'ns_1@172.23.121.78' saw that node 'ns_1@172.23.121.74' came up. Tags: []"} 2021-01-30 03:59:13,591 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.106.233', u'tstamp': 1612007948139L, u'shortText': u'node up', u'serverTime': u'2021-01-30T03:59:08.139Z', u'text': u"Node 'ns_1@172.23.106.233' saw that node 'ns_1@172.23.121.74' came up. Tags: []"} 2021-01-30 03:59:13,591 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.105.175', u'tstamp': 1612007948006L, u'shortText': u'node up', u'serverTime': u'2021-01-30T03:59:08.006Z', u'text': u"Node 'ns_1@172.23.105.175' saw that node 'ns_1@172.23.121.74' came up. Tags: []"} 2021-01-30 03:59:13,591 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 0, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.105.175', u'tstamp': 1612007947645L, u'shortText': u'message', u'serverTime': u'2021-01-30T03:59:07.645Z', u'text': u"Started node add transaction by adding node 'ns_1@172.23.121.74' to nodes_wanted (group: undefined)"} 2021-01-30 03:59:13,592 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.106.236', u'tstamp': 1612007244833L, u'shortText': u'node down', u'serverTime': u'2021-01-30T03:47:24.833Z', u'text': u"Node 'ns_1@172.23.106.236' saw that node 'ns_1@172.23.106.238' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-01-30 03:59:13,592 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.121.78', u'tstamp': 1612007244832L, u'shortText': u'node down', u'serverTime': u'2021-01-30T03:47:24.832Z', u'text': u"Node 'ns_1@172.23.121.78' saw that node 'ns_1@172.23.106.238' went down. Details: [{nodedown_reason,\n connection_closed}]"} 2021-01-30 03:59:13,592 | test | ERROR | pool-1-thread-14 | [rest_client:print_UI_logs:2595] {u'code': 5, u'module': u'ns_node_disco', u'type': u'warning', u'node': u'ns_1@172.23.106.233', u'tstamp': 1612007244830L, u'shortText': u'node down', u'serverTime': u'2021-01-30T03:47:24.830Z', u'text': u"Node 'ns_1@172.23.106.233' saw that node 'ns_1@172.23.106.238' went down. Details: [{nodedown_reason,\n connection_closed}]"} Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2593 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2593 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 Traceback (most recent call last): File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) File "/opt/jython/Lib/logging/__init__.py", line 874, in emit stream.write(fs % msg) NullPointerException: java.lang.NullPointerException Logged from file rest_client.py, line 2595 2021-01-30 03:59:13,663 | test | ERROR | pool-1-thread-14 | [rest_client:add_node:1152] Add_node error: ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] 2021-01-30 03:59:13,664 | test | ERROR | pool-1-thread-14 | [task:call:238] Error adding node: 172.23.121.74 to the cluster:172.23.105.175 - ["Join completion call failed. Got HTTP status 500 from REST call post to http://172.23.121.74:8091/completeJoin. Body was: \"[\\\"Unexpected server error, request logged.\\\"]\""] 2021-01-30 03:59:20,456 | test | INFO | MainThread | [bucket_ready_functions:perform_tasks_from_spec:4749] Done Performing scope/collection specific operations FAIL 2021-01-30 03:59:20,766 | test | INFO | MainThread | [Collections:tearDown:74] Printing bucket stats before teardown 2021-01-30 03:59:20,913 | test | INFO | MainThread | [table_view:display:72] Bucket statistics +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | Bucket | Type | Replicas | Durability | TTL | Items | RAM Quota | RAM Used | Disk Used | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ | bucket1 | couchbase | 1 | none | 0 | 69996 | 419430400 | 100849728 | 228610659 | | bucket2 | couchbase | 2 | none | 0 | 59628 | 1258291200 | 135603152 | 141153051 | | bucket3 | ephemeral | 1 | none | 0 | 55104 | 419430400 | 93148016 | 136 | | default | couchbase | 1 | none | 0 | 60294576 | 71303168000 | 24238354328 | 27597195432 | +---------+-----------+----------+------------+-----+----------+-------------+-------------+-------------+ 2021-01-30 03:59:26,092 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.105.175: No 'sanitizers.log.*' files found 2021-01-30 03:59:30,749 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.106.233: No 'sanitizers.log.*' files found 2021-01-30 03:59:35,131 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.106.236: No 'sanitizers.log.*' files found 2021-01-30 03:59:38,171 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.106.238: No 'sanitizers.log.*' files found 2021-01-30 03:59:38,977 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.106.250: No 'sanitizers.log.*' files found 2021-01-30 03:59:39,739 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.106.251: No 'sanitizers.log.*' files found 2021-01-30 03:59:40,561 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.121.74: No 'sanitizers.log.*' files found 2021-01-30 03:59:43,881 | test | WARNING | MainThread | [basetestcase:check_coredump_exist:756] 172.23.121.78: No 'sanitizers.log.*' files found ====================================================================== FAIL: test_volume_taf (volumetests.Collections.volume) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/volumetests/Collections.py", line 406, in test_volume_taf self.wait_for_rebalance_to_complete(rebalance_task) File "pytests/volumetests/Collections.py", line 236, in wait_for_rebalance_to_complete self.assertTrue(task.result, "Rebalance Failed") AssertionError: Rebalance Failed ---------------------------------------------------------------------- Ran 1 test in 4681.998s FAILED (failures=1) During the test, Remote Connections: 329, Disconnections: 329 SDK Connections: 28, Disconnections: 0 !!!!!! CRITICAL :: SDK disconnection mismatch !!!!! summary so far suite volumetests.Collections.volume , pass 0 , fail 1 failures so far... volumetests.Collections.volume.test_volume_taf testrunner logs, diags and results are available under /data/workspace/temp_durability_volume/logs/testrunner-21-Jan-30_02-41-38/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.2.2/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 1h 18m 19s 2 actionable tasks: 2 executed Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 7.0.0-4342,product=cb,parallel=True' [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE