Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s71801-32g-12c (TEMPP0 jython_slave slv-s71801 PY3P0 bkrs3) in workspace /data/workspace/temp_collections_e2e_app Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_collections_e2e_app # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision dc19124d6ebfbb427784da68cb72127dc05fbb48 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f dc19124d6ebfbb427784da68cb72127dc05fbb48 > /usr/bin/git rev-list 028f0c5e57f1243d52f1e1326547723daba6da57 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #11 jenkins-temp_collections_e2e_app-11 # timeout=10 [temp_collections_e2e_app] $ /bin/bash -x /tmp/jenkins614472490281443422.sh + jython_bin=/opt/jython/bin + jython_path=/opt/jython/bin/jython + ini_file=/tmp/temp_collections_e2e_app.ini + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + echo ' [global] username:Administrator password:Membase123 port:8091 n1ql_port:8093 index_port:9102 # data_path:/data/dta # index_path:/data/idx # cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [servers] 1:_1 2:_2 3:_3 4:_4 5:_5 6:_6 7:_7 8:_8 9:_9 10:_10 [_1] ip:172.23.107.126 [_2] ip:172.23.107.127 [_3] ip:172.23.107.129 [_4] ip:172.23.107.131 [_5] ip:172.23.107.142 [_6] ip:172.23.106.116 [_7] ip:172.23.104.247 [_8] ip:172.23.105.1 [_9] ip:172.23.105.183 [_10] ip:172.23.105.137 [membase] rest_username:Administrator rest_password:password ' + echo 'bucket_collections.travel_sample_app.TravelSampleApp: run_app,playbook=rebalance ' + rm -rf 'logs/*' jars + '[' '' '!=' '' ']' + sdk_client_params='-P transaction_version=1.1.6 -P client_version=default' + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + '[' false == true ']' + rm -f /tmp/test.pcap + '[' false == true ']' + '[' '-p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True' == '' ']' + test_params='-p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219' + echo ' activities: - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "eventing" - type: "cluster" op_type: "rebalance_out" service: "kv" - type: "cluster" op_type: "rebalance_in" nodes: 2 service: "kv,kv" - type: "cluster" op_type: "rebalance_out" service: "kv" - type: "cluster" op_type: "rebalance_swap" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "cbas" - type: "cluster" op_type: "rebalance_in" nodes: 2 service: "cbas,kv" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_in" nodes: 1 service: "n1ql:index" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "n1ql:index" - type: "cluster" op_type: "rebalance_swap" nodes: 1 service: "n1ql:index" - type: "cluster" op_type: "rebalance_in" nodes: 1 service: "eventing" ' + echo ' cb_cluster: nodes_init: 12 services: - "kv" - "kv" - "kv" - "index,n1ql" - "index,n1ql" - "backup" - "cbas" - "cbas" - "eventing" - "eventing" - "fts" - "fts" settings: - name: "memory_quota" memoryQuota: 3072 indexMemoryQuota: 2048 ftsMemoryQuota: 2048 cbasMemoryQuota: 2048 eventingMemoryQuota: 2048 rbac_users: - user_name: "cluster_admin" password: "cluster_admin" roles: - "cluster_admin" - user_name: "rbac_admin" password: "rbac_admin" roles: - "security_admin_local" ' + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.6 -P client_version=default -P 'args=-i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest' Starting a Gradle Daemon, 119 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.6 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_collections_e2e_app/build/classes/java/main:/data/workspace/temp_collections_e2e_app/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.3/fd95111700c087837c8276b387493699cb6a9775/java-client-3.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_collections_e2e_app/build/classes/java/main:/data/workspace/temp_collections_e2e_app/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.3/fd95111700c087837c8276b387493699cb6a9775/java-client-3.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_collections_e2e_app/build/classes/java/main:/data/workspace/temp_collections_e2e_app/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.3/fd95111700c087837c8276b387493699cb6a9775/java-client-3.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_collections_e2e_app/build/classes/java/main:/data/workspace/temp_collections_e2e_app/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.3/fd95111700c087837c8276b387493699cb6a9775/java-client-3.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_collections_e2e_app/build/classes/java/main:/data/workspace/temp_collections_e2e_app/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.6/52b93015952584c4a388372340e32147aa30f816/couchbase-transactions-1.1.6.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.3/fd95111700c087837c8276b387493699cb6a9775/java-client-3.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.3/8f01208d3fce6c44a17ee1f6d4603e4c0c618554/core-io-2.1.3.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.3/df23dbdf95f892f7a04292d040fd8b308bd66602/reactor-core-3.4.3.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/temp_collections_e2e_app.ini -c /tmp/test_run.conf -p rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: /tmp/test_run.conf Prefix: bucket_collections.travel_sample_app.TravelSampleApp Global Test input params: {'cluster_name': 'temp_collections_e2e_app', 'conf_file': '/tmp/test_run.conf', 'ini': '/tmp/temp_collections_e2e_app.ini', 'iteration': '1000', 'num_nodes': 10, 'rerun': 'False', 'skip_cleanup': 'True', 'skip_setup_cleanup': 'True', 'spec': 'test_run', 'upgrade_version': '7.0.0-5219'} Logs will be stored at /data/workspace/temp_collections_e2e_app/logs/testrunner-21-May-25_05-14-12/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/temp_collections_e2e_app.ini rerun=False,iteration=1000,skip_setup_cleanup=True,skip_cleanup=True,upgrade_version=7.0.0-5219 -t bucket_collections.travel_sample_app.TravelSampleApp.run_app,playbook=rebalance' Test Input params: {'cluster_name': 'temp_collections_e2e_app', 'ini': '/tmp/temp_collections_e2e_app.ini', 'conf_file': '/tmp/test_run.conf', 'upgrade_version': '7.0.0-5219', 'spec': 'test_run', 'rerun': 'False', 'num_nodes': 10, 'logs_folder': '/data/workspace/temp_collections_e2e_app/logs/testrunner-21-May-25_05-14-12/test_1', 'skip_cleanup': 'True', 'iteration': '1000', 'case_number': 1, 'skip_setup_cleanup': 'True', 'playbook': 'rebalance'} run_app (bucket_collections.travel_sample_app.TravelSampleApp) ... 2021-05-25 05:14:32,394 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-05-25 05:14:33,878 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= AppBase setup started for test #1 run_app ========= 2021-05-25 05:14:34,153 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= AppBase setup complete for test #1 run_app ========= 2021-05-25 05:14:34,154 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-05-25 05:14:34,426 | test | INFO | MainThread | [basetestcase:log_setup_status:545] ========= TravelSampleApp setup complete for test #1 run_app ========= 2021-05-25 05:14:34,788 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:14:34,793 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 1 #### 2021-05-25 05:14:34,796 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:14:35,229 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:14:35,563 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 42.9657160953 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.02 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 59.8433333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.9601146724 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.42162630623 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.93653544882 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 27.7078715355 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 19.2436540609 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 5.56851895063 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 6.29875325022 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:14:35,611 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:14:41,801 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:14:46,927 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:14:52,049 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.03 2021-05-25 05:14:57,157 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.06 2021-05-25 05:15:02,260 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 05:15:12,461 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:15:12,598 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 37.0420000553 sec 2021-05-25 05:15:12,602 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:15:12,796 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:15:13,023 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 79.0096501608 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.11835197253 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 98.0168931326 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 99.6059823614 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 18.3009150458 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 10.5714904752 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 8.54 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 22.292590247 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 44.5314843828 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:15:13,062 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:15:18,181 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 3.81 2021-05-25 05:15:23,302 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 8.09 2021-05-25 05:15:28,413 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 12.4 2021-05-25 05:15:33,526 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 15.86 2021-05-25 05:15:38,642 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 20.58 2021-05-25 05:15:43,750 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 25.01 2021-05-25 05:15:48,865 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 29.12 2021-05-25 05:15:53,977 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.02 2021-05-25 05:15:59,095 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:16:04,203 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:16:09,318 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:16:14,411 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:16:19,505 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 77.78 2021-05-25 05:16:29,644 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:16:29,786 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 76.7680001259 sec 2021-05-25 05:16:29,789 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:16:30,013 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:16:39,479 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.63810142826 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 77.1616666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.2566666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.21630585137 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.26202976816 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 14.6857342867 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 17.8133333333 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 35.3388220393 | Cluster node | | 172.23.107.131 | ['kv'] | | | <--- IN --- | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:16:39,519 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:16:44,716 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.27 2021-05-25 05:16:49,835 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 6.97 2021-05-25 05:16:54,954 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 10.32 2021-05-25 05:17:00,118 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.15 2021-05-25 05:17:05,217 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 18.92 2021-05-25 05:17:10,332 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 23.48 2021-05-25 05:17:15,459 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 26.09 2021-05-25 05:17:20,584 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 29.44 2021-05-25 05:17:25,687 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 33.06 2021-05-25 05:17:30,792 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 37.44 2021-05-25 05:17:35,937 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:17:41,038 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:17:46,135 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:17:56,269 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:17:56,325 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 76.8489999771 sec 2021-05-25 05:17:56,328 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:17:56,510 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:17:56,744 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 92.1865364423 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.06316140322 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 94.8952552372 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 90.8836372121 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 20.7843071898 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 11.8551554283 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 23.6733333333 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 21.1983333333 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 69.5566666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 92.9183333333 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:17:56,776 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:18:01,898 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 6.8 2021-05-25 05:18:07,026 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 13.75 2021-05-25 05:18:12,148 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 20.54 2021-05-25 05:18:17,273 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 27.26 2021-05-25 05:18:22,398 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 35.46 2021-05-25 05:18:27,512 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:18:32,611 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:18:37,717 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:18:42,821 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:18:47,921 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:18:58,052 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:18:59,209 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 62.4700000286 sec 2021-05-25 05:18:59,211 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:18:59,408 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:19:05,049 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 73.6966666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.831708252079 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 81.2489583681 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.725 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 2.79134915209 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 8.96 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 22.1353689228 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 37.3679122637 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 70.2095034917 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 05:19:05,078 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:19:10,173 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.16 2021-05-25 05:19:15,305 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 7.89 2021-05-25 05:19:20,427 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 12.28 2021-05-25 05:19:25,551 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 17.6 2021-05-25 05:19:30,757 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 21.93 2021-05-25 05:19:35,865 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 27.02 2021-05-25 05:19:40,996 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.57 2021-05-25 05:19:46,117 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.08 2021-05-25 05:19:51,223 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:19:56,341 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:20:01,456 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:20:06,575 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:20:11,661 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:20:21,778 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:20:22,115 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 77.0659999847 sec 2021-05-25 05:20:22,118 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:20:22,312 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:20:22,523 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 44.1442642623 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.69333333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 65.2588419614 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.42475250825 | --- OUT ---> | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.80833333333 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 5.733046681 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 31.8739375354 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 18.8229215281 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 56.9690817719 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:20:22,549 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:20:27,648 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.92 2021-05-25 05:20:32,740 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:20:37,823 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:20:47,937 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:20:50,056 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 27.5390000343 sec 2021-05-25 05:20:50,061 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:20:50,237 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:21:01,970 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 79.8703246883 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.20338672311 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 61.8760312672 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 13.4623397837 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 32.2922048701 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 42.6557109285 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 7.75564630386 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 58.1396901709 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.107.127 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:21:01,996 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:21:07,089 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.48 2021-05-25 05:21:12,183 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 10.66 2021-05-25 05:21:17,316 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 16.79 2021-05-25 05:21:22,417 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 21.47 2021-05-25 05:21:27,528 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.8 2021-05-25 05:21:32,621 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 29.97 2021-05-25 05:21:37,733 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.13 2021-05-25 05:21:42,838 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 36.44 2021-05-25 05:21:47,947 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.6 2021-05-25 05:21:53,048 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:21:58,141 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 62.91 2021-05-25 05:22:03,234 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:22:08,325 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:22:13,411 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:22:23,528 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:22:23,582 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 81.6129999161 sec 2021-05-25 05:22:23,585 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:22:23,772 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:22:24,009 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 65.34 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.85512850428 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 80.1235289208 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 49.9866635548 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.67352245075 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.24484591923 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 3.6515449485 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 25.8333333333 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 25.4154235904 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 59.9940030984 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:22:24,046 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:22:29,203 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 6.09 2021-05-25 05:22:34,358 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 14.53 2021-05-25 05:22:39,565 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 23.98 2021-05-25 05:22:44,670 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 32.26 2021-05-25 05:22:49,773 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 39.77 2021-05-25 05:22:54,924 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:23:00,032 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:23:05,140 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:23:10,220 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:23:15,309 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 05:23:25,440 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:23:25,571 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 61.5670001507 sec 2021-05-25 05:23:25,573 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:23:25,749 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:23:31,302 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 78.0190660311 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.675 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 83.1230520509 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.81179686328 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.57833333333 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 9.57484100356 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 46.4332144405 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 11.0987115755 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 74.6210603632 | Cluster node | | 172.23.107.129 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:23:31,329 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:23:36,428 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:23:41,513 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:23:46,598 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:23:56,721 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:23:56,778 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 25.478000164 sec 2021-05-25 05:23:56,780 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:23:56,954 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:23:57,176 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 67.3200326683 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.39840664011 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 54.0352690179 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 19.3282921862 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.76162063966 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.95496741721 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 7.31833333333 | --- OUT ---> | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 31.718861981 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 9.39968667711 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 72.6203155162 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:23:57,201 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:24:02,322 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 05:24:07,404 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:12,486 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:17,572 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:22,654 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:27,734 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:32,816 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:37,898 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:42,983 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:24:48,065 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 05:24:53,151 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 05:24:58,233 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 05:25:03,315 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 05:25:08,404 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 05:25:13,483 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 05:25:18,563 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 05:25:23,648 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 05:25:28,742 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 05:25:33,822 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.5 2021-05-25 05:25:38,911 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 05:25:43,999 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 05:25:49,084 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-05-25 05:25:54,170 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:25:59,256 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:26:04,345 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 05:26:14,595 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:26:15,546 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 138.374000072 sec 2021-05-25 05:26:15,549 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:26:15,717 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:26:21,615 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 59.401980066 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.17354490299 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 66.7716666667 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 26.0174660845 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.80333333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.455 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 19.1926935769 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 9.34682244704 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 50.9130971856 | Cluster node | | 172.23.105.137 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:26:21,644 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:26:26,739 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 05:26:31,825 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:26:36,914 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:26:42,006 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 05:26:47,095 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 05:26:52,190 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 05:26:57,270 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 05:27:02,357 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 05:27:07,447 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-05-25 05:27:12,529 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 05:27:17,611 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:27:22,693 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:27:27,775 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:27:37,891 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:27:38,009 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 76.3949999809 sec 2021-05-25 05:27:38,010 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:27:38,193 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:27:43,296 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 70.8611810197 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.44342405707 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 67.5011250188 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.88647045098 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.25684189473 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 36.1731913404 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 46.1457690962 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 11.195 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 68.0494674911 | Cluster node | | 172.23.107.129 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 05:27:43,322 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:27:48,421 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:27:53,515 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:27:58,601 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:28:03,687 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 67.95 2021-05-25 05:28:08,779 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.76 2021-05-25 05:28:13,875 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.9 2021-05-25 05:28:18,967 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.9 2021-05-25 05:28:24,051 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:28:29,138 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:28:34,226 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:28:39,315 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:28:44,405 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 67.68 2021-05-25 05:28:49,492 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 05:28:54,573 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 05:28:59,657 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:29:04,743 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:29:09,829 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:29:19,944 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:29:20,000 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 96.7070000172 sec 2021-05-25 05:29:34,220 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 2 #### 2021-05-25 05:29:34,221 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:29:34,404 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:29:34,615 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 65.1033333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.04505075085 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 62.8389526841 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 8.15292568705 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.55170919515 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.48 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 19.1936532276 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 50.5716666667 | Cluster node | | 172.23.105.183 | eventing | 7.0.0-5219-enterprise | 4.55984800507 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 78.1840016012 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:29:34,640 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:29:39,747 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:29:44,834 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:29:49,924 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:29:55,006 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 63.57 2021-05-25 05:30:00,098 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.42 2021-05-25 05:30:05,187 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 74.57 2021-05-25 05:30:10,273 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 05:30:20,388 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:30:20,510 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 45.8980000019 sec 2021-05-25 05:30:20,513 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:30:20,700 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:30:20,918 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 86.1766767936 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.42162630623 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 95.7467407815 | --- OUT ---> | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 41.2209796503 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.92678211304 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 2.23996266729 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 12.5070722535 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 10.4903496783 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 84.3461909675 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:30:20,948 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:30:26,052 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 3.42 2021-05-25 05:30:31,154 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 7.7 2021-05-25 05:30:36,293 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 11.28 2021-05-25 05:30:41,417 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 15.0 2021-05-25 05:30:46,523 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 19.03 2021-05-25 05:30:51,628 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 23.06 2021-05-25 05:30:56,726 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 27.0 2021-05-25 05:31:01,845 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 30.73 2021-05-25 05:31:06,957 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 32.58 2021-05-25 05:31:12,058 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:31:17,161 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:31:22,259 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:31:27,338 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:31:32,428 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:31:42,540 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:31:42,658 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 81.743999958 sec 2021-05-25 05:31:42,663 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:31:42,834 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:31:53,134 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 98.88 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.96176602943 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 16.2236037267 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.52337538959 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.47487541874 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 23.4891163039 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 37.6816666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 89.9856436966 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:31:53,160 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:31:58,260 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 1.79 2021-05-25 05:32:03,437 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 6.35 2021-05-25 05:32:08,539 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 8.37 2021-05-25 05:32:13,680 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 12.61 2021-05-25 05:32:18,822 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 16.42 2021-05-25 05:32:23,933 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 19.78 2021-05-25 05:32:29,061 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 28.81 2021-05-25 05:32:34,184 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 32.8 2021-05-25 05:32:39,308 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 36.85 2021-05-25 05:32:44,460 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:32:49,558 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:32:54,648 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:33:04,766 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:33:04,822 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 71.6890001297 sec 2021-05-25 05:33:04,825 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:33:04,983 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:33:05,197 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 61.6923051282 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.86014300715 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 78.515 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 56.0692678211 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.7983333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.73781747883 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 11.409239384 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 23.1258958035 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 69.5671010098 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 69.6611610194 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:33:05,227 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:33:10,326 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 6.72 2021-05-25 05:33:15,431 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 16.48 2021-05-25 05:33:20,575 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 25.0 2021-05-25 05:33:25,684 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.82 2021-05-25 05:33:30,809 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:33:35,918 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:33:41,029 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:33:46,128 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:33:51,213 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:34:01,323 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:34:01,447 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 56.254999876 sec 2021-05-25 05:34:01,450 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:34:01,628 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:34:06,953 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.34333333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 58.0733333333 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 53.75 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.08673477891 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.78147759279 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 2.96838280638 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 19.5543110489 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 58.2308230823 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 66.1977699628 | Cluster node | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:34:06,984 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:34:12,092 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.99 2021-05-25 05:34:17,276 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 7.43 2021-05-25 05:34:22,437 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 12.34 2021-05-25 05:34:27,576 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 17.72 2021-05-25 05:34:32,687 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 22.63 2021-05-25 05:34:37,795 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 27.25 2021-05-25 05:34:42,917 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.46 2021-05-25 05:34:48,048 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.49 2021-05-25 05:34:53,154 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:34:58,342 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:35:03,444 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:35:08,532 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:35:13,615 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:35:23,732 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:35:24,099 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 77.1470000744 sec 2021-05-25 05:35:24,101 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:35:24,315 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:35:24,555 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 89.3568592268 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.2549457509 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 75.2650764852 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 31.9806006466 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.71677861298 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 6.04771438105 | --- OUT ---> | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 38.9587013766 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 28.1742724757 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 94.5833333333 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:35:24,638 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:35:29,828 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:35:34,927 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:35:40,012 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:35:50,154 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.247:8091 during connect (<_realsocket at 0xd type=client open_count=1 channel=[id: 0xc6503ac3, 0.0.0.0/0.0.0.0:42688] timeout=300.0>) 2021-05-25 05:35:50,220 | test | ERROR | pool-2-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.247:8091/nodes/self error [Errno 111] Connection refused Got this failure java.net.ConnectException: Connection refused: /172.23.104.247:8091 during connect (<_realsocket at 0xe type=client open_count=1 channel=[id: 0x7e79d66c, 0.0.0.0/0.0.0.0:55430] timeout=300.0>) 2021-05-25 05:35:52,237 | test | ERROR | pool-2-thread-18 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.247:8091/nodes/self error [Errno 111] Connection refused 2021-05-25 05:35:54,323 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 29.7709999084 sec 2021-05-25 05:35:54,328 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:35:54,502 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 05:36:07,397 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 70.5190173006 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.43994266762 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 70.3128385473 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 9.89650172497 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 44.5325911235 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 32.2111298145 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 26.7708923631 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 55.575 | Cluster node | | 172.23.104.247 | ['cbas'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:36:07,434 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:36:12,677 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 3.48 2021-05-25 05:36:17,776 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 11.18 2021-05-25 05:36:22,869 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 17.84 2021-05-25 05:36:27,954 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 22.1 2021-05-25 05:36:33,040 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 26.91 2021-05-25 05:36:38,130 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 31.33 2021-05-25 05:36:43,243 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 36.45 2021-05-25 05:36:48,328 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 39.3 2021-05-25 05:36:53,411 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:36:58,493 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.04 2021-05-25 05:37:03,578 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:37:08,683 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:37:13,772 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:37:23,881 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:37:23,959 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 76.5650000572 sec 2021-05-25 05:37:23,960 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:37:24,184 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 05:37:24,441 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 50.1558359306 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.48333333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 44.92 | --- OUT ---> | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 24.6895885069 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.37485417153 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.44833333333 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 26.9286535673 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 27.4733333333 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 61.9252370715 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 48.12419793 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:37:24,474 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:37:29,585 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 7.11 2021-05-25 05:37:34,706 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 15.55 2021-05-25 05:37:39,819 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 23.04 2021-05-25 05:37:45,002 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-25 05:37:50,130 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 37.96 2021-05-25 05:37:55,242 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:38:00,342 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:38:05,461 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:38:10,559 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:38:15,651 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:38:25,766 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:38:25,888 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 61.4530000687 sec 2021-05-25 05:38:25,891 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:38:26,130 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 05:38:31,858 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 66.9542391868 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.77820739309 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 13.8816666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.57648078397 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.74007900132 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 47.4482482749 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 37.0864790094 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 63.1243854064 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 79.5005168217 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:38:31,900 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:38:36,993 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:38:42,082 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:38:47,168 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:38:57,280 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:38:57,334 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 25.4769999981 sec 2021-05-25 05:38:57,335 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:38:57,523 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 05:38:57,733 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 51.1956467475 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.49149209206 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.0630156164 | --- OUT ---> | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 7.60333333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.33661105648 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.02166666667 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 31.0666666667 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 27.2395460076 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 45.6259062349 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 50.6525108752 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:38:57,763 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:39:02,858 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:39:07,943 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:39:13,020 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:39:18,101 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 05:39:28,210 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:39:28,326 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 30.5979998112 sec 2021-05-25 05:39:28,328 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:39:28,500 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 05:39:34,474 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 83.7751485198 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.00308317917 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 10.8079730676 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.81516050535 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.77255654787 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 47.2917569414 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 46.8743229505 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 81.3280221337 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 70.8071532141 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:39:34,516 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:39:39,615 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 05:39:44,694 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:39:49,773 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:39:54,855 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:39:59,936 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.61 2021-05-25 05:40:05,022 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.61 2021-05-25 05:40:10,101 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.21 2021-05-25 05:40:15,183 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 34.29 2021-05-25 05:40:20,345 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 34.82 2021-05-25 05:40:25,427 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 05:40:30,506 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 05:40:35,582 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 05:40:40,661 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.04 2021-05-25 05:40:45,737 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.04 2021-05-25 05:40:50,819 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.64 2021-05-25 05:40:55,903 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.64 2021-05-25 05:41:00,989 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 05:41:06,068 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 05:41:11,154 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 05:41:16,232 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 05:41:21,315 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 05:41:26,392 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 43.39 2021-05-25 05:41:31,476 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 43.93 2021-05-25 05:41:36,556 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 44.46 2021-05-25 05:41:41,634 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 05:41:46,720 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:41:51,805 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:41:56,881 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:42:06,987 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:42:07,108 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 152.634000063 sec 2021-05-25 05:42:07,109 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:42:07,280 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:42:13,002 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 58.1632142082 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.59654678177 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 28.1504691745 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 23.724209193 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.9914669511 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.02066762073 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 30.6266666667 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 41.1716666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 46.1399789681 | Cluster node | | 172.23.105.1 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 05:42:13,029 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:42:18,115 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:42:23,196 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:42:28,278 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:42:33,355 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 78.25 2021-05-25 05:42:38,436 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 78.28 2021-05-25 05:42:43,520 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 64.48 2021-05-25 05:42:48,601 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.86 2021-05-25 05:42:53,688 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.86 2021-05-25 05:42:58,770 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.89 2021-05-25 05:43:03,852 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 62.0 2021-05-25 05:43:08,940 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.89 2021-05-25 05:43:14,020 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:43:19,101 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:43:24,184 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:43:29,279 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.93 2021-05-25 05:43:34,450 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:43:39,543 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:43:44,627 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:43:49,707 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 05:43:59,845 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:43:59,894 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 106.896000147 sec 2021-05-25 05:44:12,381 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 3 #### 2021-05-25 05:44:12,384 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:44:12,571 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:44:12,786 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 27.8642621421 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.73487108548 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 42.5542907618 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 10.1331644473 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.0403020151 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.05538082639 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 9.32484458592 | Cluster node | | 172.23.105.1 | eventing | 7.0.0-5219-enterprise | 7.34951003266 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 62.7318634068 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 31.8727360742 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:44:12,815 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:44:17,908 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:44:23,003 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:44:28,096 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:44:33,188 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 62.82 2021-05-25 05:44:38,278 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 63.48 2021-05-25 05:44:43,361 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 63.68 2021-05-25 05:44:48,446 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 78.27 2021-05-25 05:44:58,555 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:44:58,680 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 45.8960001469 sec 2021-05-25 05:44:58,683 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:44:58,852 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:44:59,066 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 97.7067813276 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.598343305722 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 30.5210173672 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 54.7383333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.493333333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 0.651644945168 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 9.97466751108 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 97.6582943049 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 97.3166666667 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 05:44:59,095 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:45:04,191 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 3.04 2021-05-25 05:45:09,296 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 6.97 2021-05-25 05:45:14,451 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 11.13 2021-05-25 05:45:19,559 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.73 2021-05-25 05:45:24,686 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 17.58 2021-05-25 05:45:29,785 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 20.53 2021-05-25 05:45:34,880 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 23.55 2021-05-25 05:45:40,066 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 26.73 2021-05-25 05:45:45,177 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 29.95 2021-05-25 05:45:50,276 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:45:55,371 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 05:46:00,467 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-25 05:46:05,558 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:46:10,634 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:46:20,744 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:46:21,091 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 82.0280001163 sec 2021-05-25 05:46:21,094 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:46:21,266 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:46:32,943 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 92.421540359 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.59821339289 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 27.212120202 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 67.630539491 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.16340272338 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 8.7218120302 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 13.6440155341 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 93.5466613167 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:46:32,969 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:46:38,058 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.56 2021-05-25 05:46:43,174 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 5.25 2021-05-25 05:46:48,279 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 8.69 2021-05-25 05:46:53,388 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 14.5 2021-05-25 05:46:58,493 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 17.13 2021-05-25 05:47:03,602 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 20.73 2021-05-25 05:47:08,713 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 25.25 2021-05-25 05:47:13,845 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 05:47:18,940 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 35.76 2021-05-25 05:47:24,065 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:47:29,158 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:47:34,249 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:47:39,674 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 05:47:49,819 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:47:49,878 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 76.9379999638 sec 2021-05-25 05:47:49,881 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:47:50,033 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:47:50,313 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 57.2513604647 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.53301556615 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.6073970365 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 65.8072365461 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 16.0430659489 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 13.101229959 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 10.9901831697 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 80.4013143629 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 78.2826094203 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 58.6869428772 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:47:50,338 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:47:55,427 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.03 2021-05-25 05:48:00,584 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 17.18 2021-05-25 05:48:05,752 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 26.01 2021-05-25 05:48:10,877 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 35.23 2021-05-25 05:48:15,997 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:48:21,177 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:48:26,279 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:48:31,374 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:48:36,460 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:48:46,586 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:48:46,701 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 56.3929998875 sec 2021-05-25 05:48:46,704 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:48:46,885 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:48:53,256 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.96009933499 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 40.158661378 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 37.2865843512 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.41305289824 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.78653689105 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 19.9230012833 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 83.6208189591 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 83.5413923565 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 80.8053199113 | Cluster node | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:48:53,283 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:48:58,384 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 2.46 2021-05-25 05:49:03,505 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 8.03 2021-05-25 05:49:08,647 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 13.84 2021-05-25 05:49:13,759 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 19.3 2021-05-25 05:49:18,868 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 23.87 2021-05-25 05:49:23,986 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 28.44 2021-05-25 05:49:29,108 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.43 2021-05-25 05:49:34,221 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 37.65 2021-05-25 05:49:39,329 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:49:44,428 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:49:49,532 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:49:54,631 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:49:59,720 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:50:09,831 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:50:09,946 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 76.6909999847 sec 2021-05-25 05:50:09,948 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:50:10,114 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:50:10,336 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 84.8952517458 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.48825892039 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.0583333333 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 60.1829939002 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.54824086265 | --- OUT ---> | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.75 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 8.93803539882 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 62.7650014977 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 70.4737421961 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:50:10,365 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:50:15,459 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:50:20,552 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:50:25,684 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 05:50:35,819 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:50:37,605 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 27.2710001469 sec 2021-05-25 05:50:37,607 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:50:37,845 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:50:49,367 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 39.3586833144 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 12.7093645318 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 27.0583333333 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 21.7726075797 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 64.14 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 25.8312361461 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 43.6558781929 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 37.915 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:50:49,400 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:50:54,496 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 3.54 2021-05-25 05:50:59,638 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 12.87 2021-05-25 05:51:04,747 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 17.73 2021-05-25 05:51:09,845 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 22.59 2021-05-25 05:51:14,937 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 26.89 2021-05-25 05:51:20,036 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.25 2021-05-25 05:51:25,147 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.13 2021-05-25 05:51:30,226 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 37.1 2021-05-25 05:51:35,332 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:51:40,421 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 05:51:45,507 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 05:51:50,595 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:51:55,688 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:52:00,845 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 05:52:10,976 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:52:11,036 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 81.6700000763 sec 2021-05-25 05:52:11,038 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:52:11,234 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:52:11,459 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 57.1592859881 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.79152709031 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 16.2233333333 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 67.497750075 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.711551902 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 10.4683333333 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 28.6284820448 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 66.5105581574 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 79.5316666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 82.6577552585 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:52:11,490 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:52:16,595 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 8.9 2021-05-25 05:52:21,730 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 18.44 2021-05-25 05:52:26,855 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 27.58 2021-05-25 05:52:32,013 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 37.03 2021-05-25 05:52:37,121 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:52:42,223 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 05:52:47,326 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:52:52,418 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:52:57,513 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 05:53:07,632 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.107.131:8091 during connect (<_realsocket at 0xf type=client open_count=1 channel=[id: 0xb7aff51f, 0.0.0.0/0.0.0.0:47926] timeout=300.0>) 2021-05-25 05:53:07,703 | test | ERROR | pool-2-thread-2 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.131:8091/nodes/self error [Errno 111] Connection refused 2021-05-25 05:53:11,729 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 60.2739999294 sec 2021-05-25 05:53:11,730 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:53:11,904 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:53:18,680 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 57.3430885637 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.4318810627 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.3928202393 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 32.4454963664 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.55154080765 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.90339838997 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 15.211159628 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 48.5416666667 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 50.1816666667 | Cluster node | | 172.23.107.131 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:53:18,710 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:53:23,811 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:53:28,914 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:53:34,005 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:53:44,125 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:53:44,180 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 25.503000021 sec 2021-05-25 05:53:44,183 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:53:44,335 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:53:44,553 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 67.525 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.31008850148 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 28.905 | --- OUT ---> | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 15.8344721843 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.1513949535 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.57629491367 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 19.3736562276 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 64.1172848751 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 57.7362201443 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 23.0733333333 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:53:44,578 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:53:49,674 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 05:53:54,763 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:53:59,844 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:54:04,940 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:54:10,023 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:15,098 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:20,178 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:25,256 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:30,336 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:35,415 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.58 2021-05-25 05:54:40,493 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 32.37 2021-05-25 05:54:45,569 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 34.74 2021-05-25 05:54:50,648 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 34.74 2021-05-25 05:54:55,732 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 37.11 2021-05-25 05:55:00,812 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 37.11 2021-05-25 05:55:05,894 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 39.47 2021-05-25 05:55:10,971 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.26 2021-05-25 05:55:16,049 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.84 2021-05-25 05:55:21,128 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.63 2021-05-25 05:55:26,207 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 05:55:31,283 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:55:36,364 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:55:41,467 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:55:51,585 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:55:51,696 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 127.148000002 sec 2021-05-25 05:55:51,698 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 05:55:51,951 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:55:57,220 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 36.175460818 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.2699621673 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 10.86 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.394960084 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.93653544882 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 3.22516125806 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 48.1271320409 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 22.6062898952 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.63215704604 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 05:55:57,246 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:56:02,345 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 05:56:07,426 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:56:12,503 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:56:17,582 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 05:56:22,664 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 05:56:27,743 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.26 2021-05-25 05:56:32,825 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.91 2021-05-25 05:56:37,911 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 05:56:42,990 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 05:56:48,071 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.83 2021-05-25 05:56:53,144 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.83 2021-05-25 05:56:58,230 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.78 2021-05-25 05:57:03,313 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.78 2021-05-25 05:57:08,401 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.74 2021-05-25 05:57:13,486 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.74 2021-05-25 05:57:18,565 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.7 2021-05-25 05:57:23,644 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 05:57:28,717 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:57:33,796 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:57:38,878 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 05:57:48,986 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:57:49,108 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 111.890000105 sec 2021-05-25 05:57:49,111 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 05:57:49,292 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:57:54,726 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 41.9029048548 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.09323177947 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 20.0752154957 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 14.3235720595 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.11 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.84896994708 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 43.8540642344 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 28.6183333333 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.7285121419 | Cluster node | | 172.23.105.137 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 05:57:54,750 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:57:59,855 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:58:04,940 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:58:10,026 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:58:15,130 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 75.26 2021-05-25 05:58:20,298 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 77.5 2021-05-25 05:58:25,380 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.9 2021-05-25 05:58:30,461 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.92 2021-05-25 05:58:35,542 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.92 2021-05-25 05:58:40,624 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.94 2021-05-25 05:58:45,711 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 70.11 2021-05-25 05:58:50,796 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.94 2021-05-25 05:58:55,884 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:59:00,971 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:59:06,069 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:59:11,153 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 05:59:16,243 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:59:21,329 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:59:26,421 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 05:59:36,526 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 05:59:36,576 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 101.855999947 sec 2021-05-25 05:59:46,552 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 4 #### 2021-05-25 05:59:46,552 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 05:59:46,716 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 05:59:46,907 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 23.6729387844 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.10836847281 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.07180119669 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 9.71032367746 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.29826169564 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 2.32193232844 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 7.72871976932 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 21.7641962143 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 20.648677478 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 7.92768429487 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 05:59:46,940 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 05:59:52,046 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 05:59:57,132 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:00:02,223 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:00:07,311 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 61.6 2021-05-25 06:00:12,394 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.32 2021-05-25 06:00:17,483 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 77.08 2021-05-25 06:00:22,657 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:00:32,773 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:00:32,895 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 45.9919998646 sec 2021-05-25 06:00:32,898 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:00:33,085 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:00:33,296 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 90.1667529086 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.67518917297 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 2.44504075068 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.64491775411 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.29800971843 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 57.5533333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 97.4733754437 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 92.8772931377 | --- OUT ---> | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 19.5293097988 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:00:33,319 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:00:38,431 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 3.89 2021-05-25 06:00:43,584 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 9.34 2021-05-25 06:00:48,749 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 10.28 2021-05-25 06:00:53,846 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 13.94 2021-05-25 06:00:58,969 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 14.67 2021-05-25 06:01:04,107 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 19.68 2021-05-25 06:01:09,283 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 24.63 2021-05-25 06:01:14,398 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 28.83 2021-05-25 06:01:19,515 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 32.9 2021-05-25 06:01:24,617 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:01:29,720 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:01:34,822 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:01:39,914 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:01:45,003 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:01:55,111 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:01:55,221 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 81.9289999008 sec 2021-05-25 06:01:55,224 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:01:55,415 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:02:06,532 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 65.9355677405 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.93006550109 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.5797903368 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.16011933532 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.18821467323 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 60.78 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 36.8460463259 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.5733333333 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:02:06,555 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:02:11,658 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 1.73 2021-05-25 06:02:16,786 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 5.58 2021-05-25 06:02:21,946 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 9.12 2021-05-25 06:02:27,094 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 14.82 2021-05-25 06:02:32,217 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 19.98 2021-05-25 06:02:37,424 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 28.43 2021-05-25 06:02:42,561 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.09 2021-05-25 06:02:47,688 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 37.17 2021-05-25 06:02:52,832 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:02:57,946 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:03:03,033 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:03:13,163 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:03:13,223 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 66.6940000057 sec 2021-05-25 06:03:13,226 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:03:13,391 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:03:13,612 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 62.421868489 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.63634545515 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 26.85 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 58.5542409266 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 18.7300739455 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 19.9424225783 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 40.1816666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 45.0179450797 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 50.625020834 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 13.1235520592 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:03:13,638 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:03:18,750 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 6.8 2021-05-25 06:03:23,894 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 14.22 2021-05-25 06:03:29,134 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 23.04 2021-05-25 06:03:34,269 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.74 2021-05-25 06:03:39,394 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:03:44,506 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:03:49,621 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:03:54,717 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:03:59,802 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:04:09,926 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:04:11,559 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 57.9519999027 sec 2021-05-25 06:04:11,562 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:04:11,736 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:04:18,618 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.01 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 25.13 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 58.1780607313 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 17.1616666667 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.34682244704 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 49.4024900415 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 50.4175069584 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 46.5384487184 | --- OUT ---> | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.58532391874 | Cluster node | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:04:18,642 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:04:23,776 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 1.47 2021-05-25 06:04:28,910 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 7.27 2021-05-25 06:04:34,065 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 12.73 2021-05-25 06:04:39,213 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 17.83 2021-05-25 06:04:44,420 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 22.93 2021-05-25 06:04:49,592 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 28.62 2021-05-25 06:04:54,740 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 32.96 2021-05-25 06:04:59,865 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 37.48 2021-05-25 06:05:05,002 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:05:10,118 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:05:15,226 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:05:20,328 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:05:25,417 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:05:35,530 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:05:35,647 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 77.0310001373 sec 2021-05-25 06:05:35,648 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:05:35,832 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:05:36,052 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 81.5867663751 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.36342272371 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 33.7000066769 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 69.2490026207 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 12.863118948 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 11.9503983466 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 48.7241879302 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 72.0164265563 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 18.877925195 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:05:36,084 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:05:41,200 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:05:46,275 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:05:51,359 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:06:01,473 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:06:02,144 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 26.0980000496 sec 2021-05-25 06:06:02,147 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:06:02,315 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:06:13,835 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 23.1253854231 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.29864826575 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.8104270142 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 38.1608159314 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 47.7590546194 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 10.5877067456 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 23.4887248121 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 11.5038646055 | Cluster node | | 172.23.104.247 | ['cbas'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:06:13,867 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:06:18,974 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 2.2 2021-05-25 06:06:24,127 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 8.27 2021-05-25 06:06:29,252 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 16.85 2021-05-25 06:06:34,398 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 22.05 2021-05-25 06:06:39,526 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 26.37 2021-05-25 06:06:44,642 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 31.38 2021-05-25 06:06:49,805 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 06:06:54,930 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 39.83 2021-05-25 06:07:00,032 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:07:05,121 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 69.04 2021-05-25 06:07:10,214 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:07:15,305 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:07:20,421 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 06:07:30,552 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:07:30,605 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 76.7720000744 sec 2021-05-25 06:07:30,607 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:07:30,755 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:07:30,973 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 68.7498958385 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.86333333333 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.7051335386 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 72.2119439305 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.05016833894 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.90660155664 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 58.88 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 67.5508811636 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 72.8128802147 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.2735623003 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:07:31,000 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:07:36,115 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 7.26 2021-05-25 06:07:41,263 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 16.8 2021-05-25 06:07:46,398 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 26.79 2021-05-25 06:07:51,536 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 35.77 2021-05-25 06:07:56,671 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:08:01,786 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:08:06,904 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:08:12,009 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:08:17,104 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:08:27,223 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:08:27,582 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 56.6150000095 sec 2021-05-25 06:08:27,586 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:08:27,775 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:08:34,756 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 35.7811927064 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.31833333333 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.25166666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 33.1265241121 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.3147069591 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 2.94338238971 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 12.2324975795 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 24.3491883604 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.41915030629 | Cluster node | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:08:34,783 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:08:39,888 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 06:08:44,980 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:08:50,062 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:09:00,221 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:09:00,280 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 25.5259997845 sec 2021-05-25 06:09:00,283 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:09:00,444 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:09:00,642 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 14.9176611774 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.363333333333 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.1665474031 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.952282538 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.834986083565 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.01668361139 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 5.53632642194 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 7.08345139086 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 27.9437990633 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.74660844639 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 06:09:00,674 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:09:05,776 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:09:10,857 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:09:15,940 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:09:21,025 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 06:09:31,144 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:09:31,260 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 30.6229999065 sec 2021-05-25 06:09:31,263 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:09:31,450 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:09:36,875 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 32.9666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.01006683445 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 4.09159847336 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.529437215 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.34325522483 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 6.24022364962 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 14.7909004676 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 25.1020850348 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 3.95023535086 | --- OUT ---> | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:09:36,904 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:09:42,009 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:09:47,092 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:09:52,187 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.76 2021-05-25 06:09:57,279 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 32.65 2021-05-25 06:10:02,368 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 35.29 2021-05-25 06:10:07,460 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 35.29 2021-05-25 06:10:12,556 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 37.94 2021-05-25 06:10:17,644 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 39.71 2021-05-25 06:10:22,739 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.59 2021-05-25 06:10:27,826 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 43.24 2021-05-25 06:10:32,910 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 43.24 2021-05-25 06:10:37,999 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 06:10:43,091 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:10:48,233 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:10:53,319 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:11:03,430 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:11:03,545 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 86.6719999313 sec 2021-05-25 06:11:03,546 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:11:03,752 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:11:09,470 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 35.5727404543 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.416652778241 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 13.0468841147 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 17.1635562381 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.35662144595 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.66988867409 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 12.3933333333 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 12.5808279448 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 28.7233333333 | Cluster node | | 172.23.107.131 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+----------------+--------------+ 2021-05-25 06:11:09,496 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:11:14,598 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:11:19,681 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:11:24,766 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:11:29,858 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 64.53 2021-05-25 06:11:34,947 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.91 2021-05-25 06:11:40,035 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 73.59 2021-05-25 06:11:45,121 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 06:11:50,211 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:11:55,311 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:12:00,401 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:12:05,486 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:12:10,573 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:12:15,655 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:12:20,736 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:12:25,819 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:12:30,901 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 74.76 2021-05-25 06:12:35,984 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:12:46,098 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:12:46,148 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 96.6840000153 sec 2021-05-25 06:12:57,095 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 5 #### 2021-05-25 06:12:57,098 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:12:57,273 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:12:57,486 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 24.4016666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.521657972367 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.2323488233 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 32.1536872272 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.96828386194 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.40833333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 3.49 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 0.914954252287 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 23.4129431176 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 9.53301556615 | --- OUT ---> | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 06:12:57,510 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:13:02,605 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:13:07,697 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:13:12,801 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:13:17,890 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 61.32 2021-05-25 06:13:22,982 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.03 2021-05-25 06:13:28,078 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 67.9 2021-05-25 06:13:33,167 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.14 2021-05-25 06:13:38,266 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 75.11 2021-05-25 06:13:43,357 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 76.91 2021-05-25 06:13:48,447 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 06:13:58,565 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:13:58,684 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 61.2000000477 sec 2021-05-25 06:13:58,687 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:13:58,851 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:13:59,053 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 67.710541691 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.06493225113 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 29.606480324 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 65.8589023516 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.65471726414 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.63348778293 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 34.6089101485 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 17.7613706438 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 60.4320029379 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:13:59,082 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:14:04,197 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 4.35 2021-05-25 06:14:09,328 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 8.93 2021-05-25 06:14:14,477 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 13.49 2021-05-25 06:14:19,665 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 17.81 2021-05-25 06:14:24,815 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 21.53 2021-05-25 06:14:29,926 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 25.2 2021-05-25 06:14:35,046 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 29.18 2021-05-25 06:14:40,167 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 32.86 2021-05-25 06:14:45,318 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:14:50,427 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:14:55,543 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:15:00,625 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:15:05,720 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:15:15,831 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:15:15,966 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 76.9179999828 sec 2021-05-25 06:15:15,969 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:15:16,151 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:15:26,901 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.06186976015 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 16.0152669211 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 70.3571190841 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.78166666667 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 6.24979167361 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 25.7933659416 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 18.49 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 66.5347955998 | Cluster node | | 172.23.107.131 | ['kv'] | | | <--- IN --- | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:15:26,931 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:15:32,069 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 1.15 2021-05-25 06:15:37,203 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 7.06 2021-05-25 06:15:42,371 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 10.57 2021-05-25 06:15:47,502 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 17.15 2021-05-25 06:15:52,631 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 21.7 2021-05-25 06:15:57,760 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 30.52 2021-05-25 06:16:02,885 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 33.86 2021-05-25 06:16:08,035 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 37.82 2021-05-25 06:16:13,165 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:16:18,266 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:16:23,391 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:16:33,588 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:16:33,641 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 66.742000103 sec 2021-05-25 06:16:33,644 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:16:33,790 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:16:34,005 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 60.5716666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.185119752 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.5847569207 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 63.75 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 18.7006526783 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 10.7314878085 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 67.840152236 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 11.0446318456 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 48.7794325651 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 51.1976897524 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:16:34,032 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:16:39,142 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 5.16 2021-05-25 06:16:44,348 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 13.59 2021-05-25 06:16:49,509 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 21.96 2021-05-25 06:16:54,634 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 29.46 2021-05-25 06:16:59,753 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 37.19 2021-05-25 06:17:04,910 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:17:10,019 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:17:15,132 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:17:20,217 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:17:25,302 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:17:35,417 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:17:35,539 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 61.5390000343 sec 2021-05-25 06:17:35,542 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:17:35,723 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:17:41,211 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 68.4094734912 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.1982300295 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.7159308701 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 60.9129695677 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 17.2427585747 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.08333333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 37.6143419911 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 14.5842707865 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 67.6033333333 | Cluster node | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:17:41,243 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:17:46,358 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.4 2021-05-25 06:17:51,484 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 8.6 2021-05-25 06:17:56,634 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 14.74 2021-05-25 06:18:01,755 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 19.71 2021-05-25 06:18:06,875 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 25.15 2021-05-25 06:18:12,040 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.76 2021-05-25 06:18:17,183 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.37 2021-05-25 06:18:22,332 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:18:27,448 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:18:32,562 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:18:37,657 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:18:42,737 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:18:47,957 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 06:18:58,092 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:18:58,207 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 76.996999979 sec 2021-05-25 06:18:58,210 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:18:58,382 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:18:58,582 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 53.0466666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.75154161528 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.0016666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.28639045365 | --- OUT ---> | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.65638384747 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 20.9049608094 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 13.6185603093 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 59.895 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 36.6937768741 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:18:58,605 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:19:03,703 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:19:08,792 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:19:13,882 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:19:23,996 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:19:24,528 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 25.9520001411 sec 2021-05-25 06:19:24,529 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:19:24,697 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:19:36,078 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 39.7153428219 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.71294768595 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.238537309 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 40.6266666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.9406039597 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 14.1373735353 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 27.5274157528 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 44.6338983616 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:19:36,105 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:19:41,312 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 1.52 2021-05-25 06:19:46,490 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 9.16 2021-05-25 06:19:51,640 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 16.84 2021-05-25 06:19:56,730 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 21.89 2021-05-25 06:20:01,851 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 27.12 2021-05-25 06:20:06,944 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 32.52 2021-05-25 06:20:12,065 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 36.77 2021-05-25 06:20:17,170 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:20:22,267 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:20:27,351 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 06:20:32,441 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:20:37,523 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:20:47,642 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:20:47,696 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 71.6210000515 sec 2021-05-25 06:20:47,698 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:20:47,848 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:20:48,048 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 62.5533333333 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.82833333333 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.7968632811 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 76.0166666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 16.6683333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 13.0485508092 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 68.3333333333 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 11.4581423643 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 62.9706171564 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 58.1013966201 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:20:48,071 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:20:53,181 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.19 2021-05-25 06:20:58,322 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 15.86 2021-05-25 06:21:03,461 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 24.14 2021-05-25 06:21:08,578 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.59 2021-05-25 06:21:13,788 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:21:18,900 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:21:24,010 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:21:29,099 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:21:34,194 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:21:39,313 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 06:21:49,459 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:21:49,624 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 61.5799999237 sec 2021-05-25 06:21:49,625 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:21:49,815 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:21:55,359 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.2850880848 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 13.4693470013 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.6439547985 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.0166666667 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 10.1313289557 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 44.6265816446 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 13.9343032848 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 58.7243120719 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 63.645 | Cluster node | | 172.23.107.142 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:21:55,384 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:22:00,502 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:22:05,588 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:22:10,670 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:22:20,786 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:22:20,844 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 25.486000061 sec 2021-05-25 06:22:20,846 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:22:21,000 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:22:21,210 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 2.29526257804 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.833305556481 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.5282078632 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 20.1523384113 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.78003900195 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 3.51660805653 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 9.97016616944 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 2.49841661389 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 23.23 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 41.77 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 06:22:21,240 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:22:26,336 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:22:31,423 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:22:36,507 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:22:41,589 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.0 2021-05-25 06:22:46,671 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.0 2021-05-25 06:22:51,752 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.0 2021-05-25 06:22:56,835 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.0 2021-05-25 06:23:01,966 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.0 2021-05-25 06:23:07,053 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.0 2021-05-25 06:23:12,144 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.0 2021-05-25 06:23:17,233 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 06:23:22,322 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 06:23:27,451 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 06:23:32,542 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 42.0 2021-05-25 06:23:37,641 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:23:42,720 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:23:47,802 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:23:57,917 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:23:58,035 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 96.8269999027 sec 2021-05-25 06:23:58,036 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:23:58,229 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:24:03,736 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 7.27871798712 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.74512483749 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 9.0334838914 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 34.8404166805 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.73984200527 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.03333333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 9.03666666667 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 35.3138228726 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 17.291908955 | Cluster node | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:24:03,757 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:24:08,865 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:24:13,953 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:24:19,036 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:24:24,128 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 06:24:29,210 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.46 2021-05-25 06:24:34,292 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 34.62 2021-05-25 06:24:39,377 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 06:24:44,467 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 06:24:49,555 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.38 2021-05-25 06:24:54,641 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 43.85 2021-05-25 06:24:59,733 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 06:25:04,834 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:25:09,920 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:25:20,076 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:25:20,953 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 77.2179999352 sec 2021-05-25 06:25:20,954 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:25:21,131 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:25:27,384 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.21666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.31830929487 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 25.3129114515 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.63666666667 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.92450377481 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 13.9585659761 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 5.055 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 23.5907863595 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 24.375 | Cluster node | | 172.23.107.142 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 06:25:27,420 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:25:32,523 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:25:37,611 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:25:42,703 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:25:47,788 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 06:25:52,881 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 75.94 2021-05-25 06:25:57,971 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 75.94 2021-05-25 06:26:03,056 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 73.25 2021-05-25 06:26:08,145 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 06:26:13,233 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:26:18,318 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:26:23,404 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:26:28,487 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:26:33,572 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:26:38,657 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:26:43,736 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:26:48,822 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:26:53,907 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:26:59,000 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:27:04,108 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:27:14,256 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:27:14,311 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 106.93200016 sec 2021-05-25 06:27:25,196 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 6 #### 2021-05-25 06:27:25,198 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:27:25,388 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:27:25,585 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | eventing | 7.0.0-5219-enterprise | 7.94333333333 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.305 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 2.57833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 23.7538123094 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.55653883972 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 0.833333333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 11.9014683089 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 9.245 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 14.036900615 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 15.4975584554 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 06:27:25,608 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:27:30,703 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:27:35,782 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:27:40,874 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:27:45,967 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 61.93 2021-05-25 06:27:51,058 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.23 2021-05-25 06:27:56,141 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.25 2021-05-25 06:28:01,229 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.28 2021-05-25 06:28:06,315 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.24 2021-05-25 06:28:11,417 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 69.29 2021-05-25 06:28:16,505 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.57 2021-05-25 06:28:21,591 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.55 2021-05-25 06:28:26,743 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.85 2021-05-25 06:28:31,891 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:28:42,040 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:28:42,154 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 76.5720000267 sec 2021-05-25 06:28:42,157 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:28:42,322 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:28:42,526 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.69005633521 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 2.57670961183 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 49.505 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.95333333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.48497525041 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 17.84 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 14.8947517541 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 76.4583333333 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 75.364178806 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:28:42,552 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:28:47,668 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 3.25 2021-05-25 06:28:52,822 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 7.62 2021-05-25 06:28:57,990 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 11.6 2021-05-25 06:29:03,109 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 15.78 2021-05-25 06:29:08,229 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 19.67 2021-05-25 06:29:13,351 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 23.74 2021-05-25 06:29:18,523 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 27.7 2021-05-25 06:29:23,641 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 31.83 2021-05-25 06:29:28,766 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:29:33,869 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:29:38,986 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:29:44,076 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:29:49,157 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:29:59,276 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:29:59,411 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 76.8900001049 sec 2021-05-25 06:29:59,414 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:29:59,654 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:30:09,937 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.545 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 4.09307915735 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 72.0062001033 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.21682028034 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.14666666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 48.645 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 2.915 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 64.74 | Cluster node | | 172.23.107.142 | ['kv'] | | | <--- IN --- | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:30:09,966 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:30:15,073 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.48 2021-05-25 06:30:20,214 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 7.31 2021-05-25 06:30:25,348 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 10.16 2021-05-25 06:30:30,482 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.74 2021-05-25 06:30:35,638 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 19.07 2021-05-25 06:30:40,762 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 25.49 2021-05-25 06:30:45,918 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 32.4 2021-05-25 06:30:51,061 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 36.26 2021-05-25 06:30:56,177 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 39.89 2021-05-25 06:31:01,309 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:31:06,398 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:31:11,553 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:31:21,678 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:31:21,733 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 71.7980000973 sec 2021-05-25 06:31:21,736 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:31:21,891 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:31:22,099 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 63.41 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.80166666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.26 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 64.2954765159 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.4829505683 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.17 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 69.7495247307 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 10.0785013084 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 65.4694244904 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 56.0942682378 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:31:22,121 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:31:27,230 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 6.33 2021-05-25 06:31:32,436 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 15.16 2021-05-25 06:31:37,586 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 23.2 2021-05-25 06:31:42,759 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 32.35 2021-05-25 06:31:47,911 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:31:53,019 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:31:58,153 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:32:03,306 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:32:08,397 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:32:18,516 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:32:19,111 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 57.0179998875 sec 2021-05-25 06:32:19,115 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:32:19,283 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:32:25,743 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.77506291772 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.85836052474 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 53.0732308923 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.7434457241 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.42666666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 53.2298923369 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 5.78019267309 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 48.6707668717 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 49.2983333333 | --- OUT ---> | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:32:25,767 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:32:30,877 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.11 2021-05-25 06:32:36,029 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 7.68 2021-05-25 06:32:41,167 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 13.31 2021-05-25 06:32:46,302 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 19.29 2021-05-25 06:32:51,434 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 24.81 2021-05-25 06:32:56,562 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 29.38 2021-05-25 06:33:01,684 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 06:33:06,809 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 36.01 2021-05-25 06:33:11,986 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.06 2021-05-25 06:33:17,144 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:33:22,257 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:33:27,381 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:33:32,471 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:33:37,563 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:33:47,676 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:33:47,793 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 82.0529999733 sec 2021-05-25 06:33:47,796 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:33:47,956 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:33:48,168 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 63.3866666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.39847994933 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.1543461955 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 45.2083333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.81867426705 | --- OUT ---> | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 11.8481358644 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 27.1095481742 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 11.5898068366 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 44.14 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:33:48,198 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:33:53,305 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.44 2021-05-25 06:33:58,463 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:34:03,549 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:34:08,634 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 83.33 2021-05-25 06:34:18,795 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:34:18,921 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 30.7540001869 sec 2021-05-25 06:34:18,923 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:34:19,096 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:34:29,631 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 34.0355672595 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 12.6552109202 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 22.7488309397 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 47.8123906195 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 13.1264478925 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 25.9629006183 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 20.1306710224 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 31.6671944532 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.107.131 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:34:29,654 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:34:34,773 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 3.22 2021-05-25 06:34:39,880 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 11.72 2021-05-25 06:34:44,980 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 21.0 2021-05-25 06:34:50,082 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.55 2021-05-25 06:34:55,210 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 29.67 2021-05-25 06:35:00,315 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.51 2021-05-25 06:35:05,403 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.45 2021-05-25 06:35:10,523 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.2 2021-05-25 06:35:15,611 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:35:20,691 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.07 2021-05-25 06:35:25,779 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:35:30,880 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:35:36,016 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 06:35:46,167 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:35:46,230 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 76.6029999256 sec 2021-05-25 06:35:46,233 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:35:46,424 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:35:46,660 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 78.7486875219 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.78828686189 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.0696310123 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 55.9916666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.82833333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.29833333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 47.270155587 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 13.8818980316 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 49.0358172636 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 64.7128429052 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:35:46,703 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:35:51,845 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 6.56 2021-05-25 06:35:57,062 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 15.23 2021-05-25 06:36:02,236 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.63 2021-05-25 06:36:07,515 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 28.82 2021-05-25 06:36:13,015 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 36.87 2021-05-25 06:36:21,496 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:36:26,641 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:36:31,790 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:36:36,923 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:36:42,042 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:36:55,201 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:36:55,339 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 68.6849999428 sec 2021-05-25 06:36:55,342 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:36:56,201 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:37:02,519 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 51.0141835697 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.03340055668 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.4283333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 31.9516666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.46684889496 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 6.19979334022 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 20.8346527558 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 11.718138031 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 43.2302256591 | Cluster node | | 172.23.107.131 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:37:02,558 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:37:07,686 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:37:15,812 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:37:25,956 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:37:26,017 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 23.5 sec 2021-05-25 06:37:26,020 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:37:26,229 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:37:26,536 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 40.6509891502 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.78666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.15027167572 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 46.7208879852 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.04505075085 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 2.94014700735 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 24.6629222846 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 10.103501725 | --- OUT ---> | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 38.255 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.20639312023 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:37:26,584 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:37:31,750 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:37:36,855 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:37:41,960 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:37:47,065 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:37:52,170 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 06:37:57,280 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 06:38:02,388 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 06:38:07,490 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 06:38:12,592 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:38:17,696 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:38:22,799 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 06:38:32,973 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:38:33,095 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 66.5640001297 sec 2021-05-25 06:38:33,098 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:38:33,273 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:38:38,448 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+----------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 25.3154008966 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.988399226615 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.46339105652 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 33.9305655094 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.28333333333 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 1.50838361279 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 21.2022732197 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 25.730808973 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 2.0315989467 | --- OUT ---> | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+----------------+--------------+ 2021-05-25 06:38:38,471 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:38:43,565 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:38:48,650 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:38:53,743 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:38:58,882 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 06:39:03,966 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 06:39:09,069 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 06:39:14,158 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 06:39:19,243 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:39:24,328 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:39:29,420 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:39:39,533 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:39:39,651 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 61.2070000172 sec 2021-05-25 06:39:39,654 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:39:39,834 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:39:45,104 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 25.545 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.27007116785 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.50422123247 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 26.1823143741 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.6967282788 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.70480983967 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.7524542894 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 5.71633023248 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 28.384279476 | Cluster node | | 172.23.107.131 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 06:39:45,128 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:39:50,240 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:39:55,336 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:40:00,430 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:40:05,535 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 68.53 2021-05-25 06:40:10,622 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.87 2021-05-25 06:40:15,713 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 06:40:20,799 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 75.7 2021-05-25 06:40:25,921 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 06:40:31,010 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:40:36,092 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:40:41,171 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:40:46,270 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:40:51,362 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:40:56,463 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:41:01,552 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 06:41:06,631 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:41:11,726 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 73.28 2021-05-25 06:41:16,805 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:41:21,888 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 06:41:32,012 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:41:32,065 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 106.963000059 sec 2021-05-25 06:41:42,778 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 7 #### 2021-05-25 06:41:42,780 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:41:43,007 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:41:43,214 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 38.1803939869 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.11326477892 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 6.90155164081 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.1712057069 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.9082030599 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 8.69666666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 12.8131197813 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 11.356477392 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 12.9716666667 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 8.935 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:41:43,239 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:41:48,339 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:41:53,434 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:41:58,529 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:42:03,618 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 62.11 2021-05-25 06:42:08,717 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 66.04 2021-05-25 06:42:13,815 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 66.07 2021-05-25 06:42:18,908 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 66.18 2021-05-25 06:42:23,996 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 69.02 2021-05-25 06:42:29,084 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:42:34,171 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:42:44,301 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:42:44,426 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 61.2160000801 sec 2021-05-25 06:42:44,428 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:42:44,655 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:42:44,836 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 54.270762154 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.18666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 23.591179559 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 90.4003213282 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.8851814197 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 5.20684022801 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 28.6426214207 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 3.69666666667 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 56.3292721545 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:42:44,891 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:42:49,997 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 4.58 2021-05-25 06:42:55,134 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 8.87 2021-05-25 06:43:00,280 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 12.34 2021-05-25 06:43:05,434 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 17.39 2021-05-25 06:43:10,569 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 21.45 2021-05-25 06:43:15,681 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 25.21 2021-05-25 06:43:20,819 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 29.15 2021-05-25 06:43:25,980 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 32.87 2021-05-25 06:43:31,091 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:43:36,209 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:43:41,334 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:43:46,431 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:43:51,532 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:44:01,651 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:44:01,767 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 76.9329998493 sec 2021-05-25 06:44:01,770 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:44:01,950 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:44:12,217 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.95476817439 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.07 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 75.0012500208 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 15.5480741988 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 7.5 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 39.9349023535 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 13.1504383479 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 76.3824539182 | Cluster node | | 172.23.107.131 | ['kv'] | | | <--- IN --- | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:44:12,250 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:44:17,384 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 1.43 2021-05-25 06:44:22,513 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 7.06 2021-05-25 06:44:27,635 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 10.26 2021-05-25 06:44:32,796 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 16.88 2021-05-25 06:44:37,948 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 21.11 2021-05-25 06:44:43,089 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 29.42 2021-05-25 06:44:48,213 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 33.25 2021-05-25 06:44:53,361 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 37.21 2021-05-25 06:44:58,473 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:45:03,602 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:45:08,687 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:45:18,805 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:45:18,865 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 66.6519999504 sec 2021-05-25 06:45:18,868 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:45:19,019 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:45:19,232 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 58.4612563962 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.43169052818 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.1250520842 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 55.3640772654 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.6011366098 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.32213277331 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 72.9839917463 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 2.21333333333 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 51.6150538351 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 59.4526575443 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:45:19,256 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:45:24,375 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 5.78 2021-05-25 06:45:29,509 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 13.28 2021-05-25 06:45:34,660 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 21.1 2021-05-25 06:45:39,890 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 29.14 2021-05-25 06:45:45,029 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.8 2021-05-25 06:45:50,144 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:45:55,256 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:46:00,364 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:46:05,459 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:46:10,549 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:46:20,661 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:46:20,832 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 61.6039998531 sec 2021-05-25 06:46:20,835 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:46:21,006 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:46:26,296 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 52.9667655589 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.56002600043 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.089644926 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 54.9484982833 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.52674211237 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 4.39840664011 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 45.5190919849 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 3.66691112741 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 46.6665555519 | Cluster node | | 172.23.105.183 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:46:26,334 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:46:31,444 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 2.35 2021-05-25 06:46:36,578 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 8.8 2021-05-25 06:46:41,720 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 14.66 2021-05-25 06:46:46,891 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 21.64 2021-05-25 06:46:52,071 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 28.1 2021-05-25 06:46:57,280 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.08 2021-05-25 06:47:02,424 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 38.77 2021-05-25 06:47:07,549 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:47:12,674 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:47:17,780 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:47:22,877 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:47:27,980 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:47:38,094 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:47:38,213 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 71.9210000038 sec 2021-05-25 06:47:38,216 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:47:38,385 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:47:38,605 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.4 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.2966666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.4071432143 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.14469517683 | Cluster node | | 172.23.104.247 | cbas | 7.0.0-5219-enterprise | 9.09272715152 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 48.5684287714 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 14.3219053651 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 84.7397456624 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 67.915 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:47:38,637 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:47:43,895 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:47:48,983 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:47:54,065 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:48:04,191 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:48:06,094 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 27.4910001755 sec 2021-05-25 06:48:06,096 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:48:06,313 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:48:17,684 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 12.8381193647 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 20.7270121169 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 60.1326688778 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 59.79 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 17.9947000883 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 16.6947217546 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 37.4483333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 61.0397186526 | Cluster node | | 172.23.107.142 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:48:17,720 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:48:22,914 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:48:28,154 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 8.5 2021-05-25 06:48:33,263 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 19.66 2021-05-25 06:48:38,362 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 24.16 2021-05-25 06:48:43,470 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 27.94 2021-05-25 06:48:48,599 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 33.0 2021-05-25 06:48:53,703 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 37.12 2021-05-25 06:48:58,793 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:49:03,921 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 06:49:09,016 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.07 2021-05-25 06:49:14,108 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:49:19,190 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:49:24,282 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:49:34,390 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:49:34,486 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 76.8080000877 sec 2021-05-25 06:49:34,489 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:49:34,664 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:49:34,917 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.4299261679 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.42491916936 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.1829939002 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 43.4892751787 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.21528101873 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 36.797720038 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 27.3712104798 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 4.81674694578 | Cluster node | | 172.23.105.183 | kv | 7.0.0-5219-enterprise | 44.294809827 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 53.381481728 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:49:34,954 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:49:40,079 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 7.58 2021-05-25 06:49:45,230 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 17.42 2021-05-25 06:49:50,368 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 27.73 2021-05-25 06:49:55,532 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 35.39 2021-05-25 06:50:00,688 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:50:05,808 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:50:10,937 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:50:16,033 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:50:21,138 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 06:50:31,266 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:50:31,582 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 56.6689999104 sec 2021-05-25 06:50:31,585 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:50:31,746 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:50:38,487 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.92678211304 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.39324344594 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.9684661411 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 56.9023849603 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.21166666667 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 58.4609743496 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 65.570541497 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 9.945 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 56.4496805112 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:50:38,555 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:50:43,657 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:50:48,743 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:50:53,831 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:51:03,944 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:51:03,997 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 25.5149998665 sec 2021-05-25 06:51:04,000 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:51:04,180 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 06:51:04,381 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.92988450192 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.96646778441 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 24.2716666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 20.2573419114 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.95321744638 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 24.9516666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 11.6763178914 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 10.9590639376 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.5133333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 45.3639348634 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:51:04,411 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:51:09,523 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:51:14,618 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:51:19,696 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 06:51:24,779 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 06:51:29,872 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 06:51:34,960 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 06:51:40,078 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 06:51:45,200 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 06:51:50,292 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:51:55,395 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:52:05,519 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:52:06,496 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 62.1169998646 sec 2021-05-25 06:52:06,500 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 06:52:06,694 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:52:13,401 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.92810239659 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.26821219646 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 18.9640517974 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 44.3329505932 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.7084618077 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 45.4247441551 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.7015274184 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.4169232049 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 49.5566666667 | Cluster node | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 06:52:13,428 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:52:18,565 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 06:52:23,678 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:52:28,776 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:52:33,864 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.67 2021-05-25 06:52:38,971 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.67 2021-05-25 06:52:44,072 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 32.78 2021-05-25 06:52:49,167 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:52:54,250 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.0 2021-05-25 06:52:59,371 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.0 2021-05-25 06:53:04,463 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.67 2021-05-25 06:53:09,552 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.33 2021-05-25 06:53:14,640 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:53:19,726 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 06:53:24,812 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.67 2021-05-25 06:53:29,901 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 43.33 2021-05-25 06:53:35,002 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 06:53:40,108 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:53:45,194 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:53:50,286 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 06:54:00,394 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.107.127:8091 during connect (<_realsocket at 0x10 type=client open_count=1 channel=[id: 0xea8b86ee, 0.0.0.0/0.0.0.0:43436] timeout=300.0>) 2021-05-25 06:54:00,453 | test | ERROR | pool-2-thread-23 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.107.127:8091/nodes/self error [Errno 111] Connection refused 2021-05-25 06:54:03,838 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 110.441999912 sec 2021-05-25 06:54:03,841 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:54:04,006 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:54:10,772 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.51328289057 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.82544127206 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 41.4583333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.24504150138 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 22.4983333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 6.0403020151 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 5.93539569305 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.97673255775 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 22.5524184139 | Cluster node | | 172.23.107.127 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 06:54:10,796 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:54:15,934 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:54:21,029 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:54:26,144 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:54:31,252 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 64.51 2021-05-25 06:54:36,352 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 76.22 2021-05-25 06:54:41,436 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 75.06 2021-05-25 06:54:46,529 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 76.84 2021-05-25 06:54:51,624 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 06:54:56,720 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:55:01,822 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:55:06,921 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:55:12,025 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:55:17,118 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 06:55:22,226 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 73.83 2021-05-25 06:55:27,313 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:55:32,398 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:55:37,490 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:55:42,581 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:55:47,697 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 06:55:57,805 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:55:57,867 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 107.097000122 sec 2021-05-25 06:56:08,484 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 8 #### 2021-05-25 06:56:08,486 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:56:08,703 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:56:09,167 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.16154730754 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.15658072365 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 11.1875271258 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 39.3332888191 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.71015700523 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 18.3856128538 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 6.82321961301 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 7.19484508547 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.17922780309 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 35.8716660224 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:56:09,190 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:56:14,283 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:56:19,384 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:56:24,596 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 06:56:29,809 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 61.43 2021-05-25 06:56:34,963 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 64.53 2021-05-25 06:56:40,121 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 65.56 2021-05-25 06:56:45,286 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.25 2021-05-25 06:56:50,424 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.32 2021-05-25 06:56:55,562 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.36 2021-05-25 06:57:00,668 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.36 2021-05-25 06:57:05,782 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 76.54 2021-05-25 06:57:10,885 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 06:57:15,993 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 06:57:26,125 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:57:26,246 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 77.0840001106 sec 2021-05-25 06:57:26,247 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 06:57:26,450 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:57:26,665 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 23.3070551176 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.9714009533 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 56.9007183214 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.19181986366 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 45.2839308632 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 31.6661388977 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 23.4903915065 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 20.9122877189 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 55.65 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:57:26,726 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:57:31,848 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 2.89 2021-05-25 06:57:37,058 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.03 2021-05-25 06:57:42,250 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 10.87 2021-05-25 06:57:47,382 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 15.69 2021-05-25 06:57:52,542 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.99 2021-05-25 06:57:57,701 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 23.41 2021-05-25 06:58:02,911 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 26.15 2021-05-25 06:58:08,138 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 29.46 2021-05-25 06:58:13,341 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.03 2021-05-25 06:58:18,463 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:58:23,611 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 06:58:28,707 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:58:33,811 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:58:38,937 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 06:58:49,049 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 06:58:49,165 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 82.501999855 sec 2021-05-25 06:58:49,168 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 06:58:49,368 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 06:59:00,049 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 23.775 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.54675911265 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.98 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 69.7401536744 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 27.7621293688 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 17.0563823936 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.4519908665 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 75.99040016 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 06:59:00,095 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 06:59:05,216 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.06 2021-05-25 06:59:10,420 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 3.29 2021-05-25 06:59:15,654 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 7.51 2021-05-25 06:59:20,950 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 9.18 2021-05-25 06:59:26,161 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 13.7 2021-05-25 06:59:31,394 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 16.19 2021-05-25 06:59:36,604 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 18.36 2021-05-25 06:59:41,730 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 21.87 2021-05-25 06:59:46,934 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 25.54 2021-05-25 06:59:52,111 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 06:59:57,259 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 35.48 2021-05-25 07:00:02,424 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 38.83 2021-05-25 07:00:07,571 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:00:12,665 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:00:17,776 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:00:27,959 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:00:28,017 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 87.9729998112 sec 2021-05-25 07:00:28,020 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:00:28,173 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:00:28,384 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 12.1629279024 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.28657855702 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 84.9494983166 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 85.8585690238 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.5103836795 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 76.5362756046 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 67.447123625 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 12.4454148472 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.875 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 80.0266666667 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:00:28,413 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:00:33,562 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.03 2021-05-25 07:00:38,760 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 13.99 2021-05-25 07:00:43,921 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 20.23 2021-05-25 07:00:49,055 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 23.12 2021-05-25 07:00:54,164 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.57 2021-05-25 07:00:59,280 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.15 2021-05-25 07:01:04,453 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:01:09,598 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:01:14,720 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:01:19,825 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:01:24,933 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:01:35,088 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:01:35,206 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 66.8240001202 sec 2021-05-25 07:01:35,210 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:01:35,385 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:01:41,782 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 13.2018866981 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 12.4985416424 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 76.9 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.84199473316 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 70.7028450474 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 50.4682413822 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 16.6624997917 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.2423080769 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 42.0429015992 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:01:41,857 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:01:46,976 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.88 2021-05-25 07:01:52,131 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 6.84 2021-05-25 07:01:57,269 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 12.16 2021-05-25 07:02:02,411 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 18.13 2021-05-25 07:02:07,604 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 22.87 2021-05-25 07:02:12,829 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 27.49 2021-05-25 07:02:18,036 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.46 2021-05-25 07:02:23,194 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 35.79 2021-05-25 07:02:28,338 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:02:33,450 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:02:38,569 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:02:43,671 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:02:48,759 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:02:53,878 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 07:03:04,033 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:03:04,151 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 82.3729999065 sec 2021-05-25 07:03:04,153 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:03:04,321 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:03:04,526 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.92663455609 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.14030467682 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 40.5716666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.3871935468 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.765 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 34.6313214517 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 1.6133871129 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.72 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 52.940882348 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:03:04,555 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:03:09,750 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:03:14,835 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 49.97 2021-05-25 07:03:19,915 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:03:25,006 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:03:30,084 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:03:40,187 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:03:40,384 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 35.8589999676 sec 2021-05-25 07:03:40,387 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:03:40,611 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:03:51,993 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 46.6141102352 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.46505775096 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 46.0934635512 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 35.3916666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.1609086212 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 2.97156761441 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.94176472549 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 50.78 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:03:52,049 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:03:57,230 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 3.15 2021-05-25 07:04:02,338 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 9.82 2021-05-25 07:04:07,453 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 15.84 2021-05-25 07:04:12,589 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 19.11 2021-05-25 07:04:17,706 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 22.57 2021-05-25 07:04:22,859 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 26.36 2021-05-25 07:04:27,957 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 30.7 2021-05-25 07:04:33,065 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 35.18 2021-05-25 07:04:38,177 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 38.96 2021-05-25 07:04:43,273 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:04:48,358 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:04:53,444 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:04:58,532 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:05:03,628 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:05:13,753 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:05:13,805 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 81.8159999847 sec 2021-05-25 07:05:13,806 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:05:13,976 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:05:14,243 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.93486883771 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.8649522508 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 53.8008966816 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 62.8145728476 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.96 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 49.1124852081 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 44.2198073269 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 12.6281228646 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.5897235046 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 44.92 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:05:14,267 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:05:19,384 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 4.76 2021-05-25 07:05:24,529 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 14.14 2021-05-25 07:05:29,716 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 22.73 2021-05-25 07:05:34,898 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 31.24 2021-05-25 07:05:40,085 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 38.05 2021-05-25 07:05:45,262 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:05:50,378 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:05:55,494 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:06:00,591 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:06:05,678 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:06:15,792 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:06:15,915 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 61.6759998798 sec 2021-05-25 07:06:15,920 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:06:16,121 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:06:22,084 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.82705513701 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.16652778241 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 76.5088254413 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.73147561748 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 65.3666666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 43.4133333333 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 15.7569292822 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.135 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 66.4499458198 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:06:22,122 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:06:27,243 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:06:32,338 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:06:37,426 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:06:47,535 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:06:47,586 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 25.5069999695 sec 2021-05-25 07:06:47,588 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:06:47,759 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:06:47,948 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.40325994567 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.66476676166 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.82379354666 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 29.7128381194 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 13.9583333333 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 47.6842052632 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.4985356763 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 12.8947850869 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 8.98348305805 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 42.787860607 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:06:47,973 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:06:53,088 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:06:58,176 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:07:03,259 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:07:08,380 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 07:07:18,500 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:07:18,634 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 30.6889998913 sec 2021-05-25 07:07:18,638 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:07:18,819 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 07:07:24,907 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.85823569607 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.0250837514 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 41.4593090115 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.36342272371 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 31.3533333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.9914001433 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 13.2564457259 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 14.5059413696 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 30.8606379787 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:07:24,964 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:07:30,108 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:07:35,200 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:07:40,302 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:07:45,388 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 07:07:50,473 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.91 2021-05-25 07:07:55,575 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.91 2021-05-25 07:08:00,704 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 07:08:05,812 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.83 2021-05-25 07:08:10,907 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.78 2021-05-25 07:08:15,993 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.78 2021-05-25 07:08:21,095 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.43 2021-05-25 07:08:26,183 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.74 2021-05-25 07:08:31,278 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.74 2021-05-25 07:08:36,410 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.7 2021-05-25 07:08:41,503 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 07:08:46,585 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:08:51,670 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:08:56,759 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:09:06,871 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:09:06,990 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 102.088000059 sec 2021-05-25 07:09:06,993 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:09:07,203 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:09:12,776 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.11 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.1687220852 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 46.198079872 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 19.2186536442 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 46.6142230963 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 16.9529437294 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.9230012833 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 24.6991069193 | Cluster node | | 172.23.105.1 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 07:09:12,801 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:09:17,913 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:09:23,007 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:09:28,115 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:09:33,244 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 63.16 2021-05-25 07:09:38,335 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:09:43,437 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 74.21 2021-05-25 07:09:48,536 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.95 2021-05-25 07:09:53,634 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:09:58,719 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:10:03,815 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:10:08,900 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:10:14,022 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:10:19,109 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:10:24,204 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:10:29,302 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:10:34,385 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:10:39,471 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:10:44,555 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.76 2021-05-25 07:10:49,645 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:10:54,739 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:11:04,862 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:11:04,920 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 112.144999981 sec 2021-05-25 07:11:16,053 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 9 #### 2021-05-25 07:11:16,056 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:11:16,242 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:11:16,461 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.77814442611 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.588359806 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.6241145391 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 38.5416666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.94495091748 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 25.3108436948 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 7.94640178661 | Cluster node | | 172.23.105.1 | eventing | 7.0.0-5219-enterprise | 6.09186972899 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.92666666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 37.3962465416 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:11:16,496 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:11:21,608 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:11:26,703 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:11:31,828 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:11:36,967 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.86 2021-05-25 07:11:42,104 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 63.03 2021-05-25 07:11:47,253 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 65.03 2021-05-25 07:11:52,375 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 65.78 2021-05-25 07:11:57,487 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.21 2021-05-25 07:12:02,588 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.59 2021-05-25 07:12:07,703 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.58 2021-05-25 07:12:12,803 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.56 2021-05-25 07:12:17,907 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.57 2021-05-25 07:12:23,016 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.12 2021-05-25 07:12:28,122 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.06 2021-05-25 07:12:33,233 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.07 2021-05-25 07:12:38,344 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 71.32 2021-05-25 07:12:43,467 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 73.38 2021-05-25 07:12:48,608 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:12:53,709 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:13:03,848 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:13:03,971 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 107.514999866 sec 2021-05-25 07:13:03,974 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:13:04,147 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:13:04,375 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.88509808497 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.10491491808 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.6633333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 54.5590906818 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.69344489075 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 50.1816606113 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 22.1887031451 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 21.5909469684 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 51.0841283608 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:13:04,427 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:13:09,545 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 2.98 2021-05-25 07:13:14,694 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 6.9 2021-05-25 07:13:19,835 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 10.47 2021-05-25 07:13:24,966 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 14.12 2021-05-25 07:13:30,153 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 17.57 2021-05-25 07:13:35,276 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.43 2021-05-25 07:13:40,421 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 25.33 2021-05-25 07:13:45,565 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 29.24 2021-05-25 07:13:50,763 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 32.78 2021-05-25 07:13:55,898 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:14:01,013 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:14:06,128 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:14:11,223 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:14:16,329 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:14:26,471 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:14:26,588 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 82.2160000801 sec 2021-05-25 07:14:26,589 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:14:26,760 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:14:37,687 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.25221681445 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.26654555757 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 21.0166666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 92.6817886369 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.69818836353 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 51.12 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.8725102922 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 93.3803335789 | Cluster node | | 172.23.104.247 | ['kv'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:14:37,723 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:14:42,848 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.15 2021-05-25 07:14:48,028 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 4.42 2021-05-25 07:14:53,161 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 9.17 2021-05-25 07:14:58,296 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 13.6 2021-05-25 07:15:03,414 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 18.45 2021-05-25 07:15:08,563 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 22.72 2021-05-25 07:15:13,720 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 31.52 2021-05-25 07:15:18,842 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 35.09 2021-05-25 07:15:23,973 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 39.67 2021-05-25 07:15:29,105 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:15:34,188 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:15:39,308 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:15:49,460 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:15:49,529 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 71.8459999561 sec 2021-05-25 07:15:49,532 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:15:49,740 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:15:49,973 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 9.94833333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.06166666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 6.88920690918 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 72.6824227474 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.001499975 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 84.0616666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 73.619560326 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 81.8742396214 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.19348655811 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 74.1538256731 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:15:50,003 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:15:55,150 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 6.72 2021-05-25 07:16:00,316 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 16.8 2021-05-25 07:16:05,461 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 26.87 2021-05-25 07:16:10,607 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.69 2021-05-25 07:16:15,762 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:16:20,882 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:16:26,000 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:16:31,098 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:16:36,213 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:16:41,546 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 07:16:51,704 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:16:51,822 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 61.8539998531 sec 2021-05-25 07:16:51,823 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:16:52,000 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:16:57,760 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.92325127915 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.80166666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.74679155972 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.1771970467 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 15.3102551709 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 70.730691023 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 51.2468641491 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.18333333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 75.1537474375 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:16:57,786 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:17:02,890 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 2.23 2021-05-25 07:17:08,025 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 8.91 2021-05-25 07:17:13,203 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 15.13 2021-05-25 07:17:18,374 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 21.17 2021-05-25 07:17:23,543 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 26.45 2021-05-25 07:17:28,687 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.91 2021-05-25 07:17:33,845 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.54 2021-05-25 07:17:39,003 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:17:44,121 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:17:49,224 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:17:54,312 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:17:59,410 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:18:04,743 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 07:18:14,890 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:18:15,006 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 77.25 sec 2021-05-25 07:18:15,007 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:18:15,214 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:18:15,437 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.09186972899 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.78009633494 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.1983333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.9416666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.40182336372 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 66.5651573395 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 45.26 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.66015533851 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 45.434514072 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:18:15,467 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:18:20,558 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 58.62 2021-05-25 07:18:25,647 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:18:30,730 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:18:35,816 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 83.33 2021-05-25 07:18:45,934 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:18:46,043 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 30.6089999676 sec 2021-05-25 07:18:46,046 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:18:46,216 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:18:57,612 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.43510558384 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.72509541826 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 13.5931067816 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 56.9809496825 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 19.4526392986 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 49.5591740138 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 12.523542059 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 52.0332655578 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:18:57,641 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:19:02,773 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 2.42 2021-05-25 07:19:07,917 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 9.14 2021-05-25 07:19:13,049 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 17.11 2021-05-25 07:19:18,168 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 24.84 2021-05-25 07:19:23,299 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 34.3 2021-05-25 07:19:28,400 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:19:33,505 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:19:38,598 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:19:43,706 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:19:48,802 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:19:58,943 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:19:59,003 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 61.3940000534 sec 2021-05-25 07:19:59,006 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:19:59,177 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:19:59,394 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.80006333439 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.913302889904 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.69993833436 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 49.2191796803 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.77037741833 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 57.1383333333 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 47.6541275688 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 43.1761745246 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.11180372679 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 53.4624422926 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:19:59,427 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:20:04,562 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 7.66 2021-05-25 07:20:09,750 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 16.25 2021-05-25 07:20:14,927 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 25.31 2021-05-25 07:20:20,085 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 34.06 2021-05-25 07:20:25,240 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:20:30,365 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:20:35,483 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:20:40,611 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:20:45,726 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:20:50,903 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 07:21:01,058 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:21:01,174 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 61.7829999924 sec 2021-05-25 07:21:01,174 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:21:01,338 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:21:07,346 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.1949300845 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.82666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.45987053401 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 77.5233333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.84013066884 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 50.206357131 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 58.775 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.18008633477 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 68.1358069762 | Cluster node | | 172.23.104.247 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:21:07,388 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:21:12,525 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:21:17,609 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:21:22,723 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:21:32,848 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:21:32,917 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 25.5759999752 sec 2021-05-25 07:21:32,921 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:21:33,144 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:21:33,411 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.06676777946 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.49509158486 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.755 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.3054564848 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.51166666667 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 10.5201753363 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 24.6366666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 29.5066666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.7018450308 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 33.9337284033 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:21:33,447 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:21:38,627 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:21:43,714 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:21:48,818 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:21:53,904 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:21:58,996 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:22:04,104 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:22:09,194 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:22:14,278 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 07:22:19,359 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 07:22:24,448 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 07:22:29,536 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 07:22:34,622 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 07:22:39,713 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 07:22:44,798 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.5 2021-05-25 07:22:49,885 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 07:22:54,976 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-05-25 07:23:00,062 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 07:23:05,170 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:23:10,262 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:23:20,384 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:23:20,499 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 107.090000153 sec 2021-05-25 07:23:20,500 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:23:20,670 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:23:26,555 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.06186976015 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.28517617254 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.7048215863 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 45.4434852172 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.40159330678 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 5.73176219604 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 16.0682391039 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 47.6282542751 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 38.8016666667 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:23:26,584 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:23:31,730 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:23:36,815 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:23:41,904 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:23:47,005 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.0 2021-05-25 07:23:52,101 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 07:23:57,194 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 07:24:02,282 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 07:24:07,387 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 07:24:12,477 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.0 2021-05-25 07:24:17,569 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.0 2021-05-25 07:24:22,664 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 07:24:27,755 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:24:32,845 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:24:37,946 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 75.0 2021-05-25 07:24:48,068 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:24:48,210 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 81.6600000858 sec 2021-05-25 07:24:48,211 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:24:48,385 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:24:54,101 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.325 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.41321955935 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.07026900897 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 30.3389887004 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.45022251113 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 18.019234295 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 29.2166666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.118870629 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 30.3916666667 | Cluster node | | 172.23.104.247 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 07:24:54,134 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:24:59,247 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:25:04,351 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:25:09,457 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:25:14,568 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 67.51 2021-05-25 07:25:19,701 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 74.61 2021-05-25 07:25:24,812 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 75.95 2021-05-25 07:25:29,967 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 73.91 2021-05-25 07:25:35,085 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:25:40,203 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:25:45,293 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:25:50,382 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.03 2021-05-25 07:25:55,473 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:26:00,562 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:26:05,654 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:26:10,763 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:26:15,867 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:26:20,963 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:26:26,068 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 76.85 2021-05-25 07:26:31,198 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:26:36,283 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:26:46,394 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:26:46,453 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 112.354000092 sec 2021-05-25 07:26:57,165 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 10 #### 2021-05-25 07:26:57,167 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:26:57,358 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:26:57,621 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.38326417012 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.14166666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.9885329511 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 29.5316666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.08840147336 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 6.3579094727 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 4.45159247346 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 13.9640690621 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.95330077832 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 16.4141582312 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:26:57,648 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:27:02,766 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:27:07,861 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:27:12,983 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:27:18,121 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 62.99 2021-05-25 07:27:23,260 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.59 2021-05-25 07:27:28,404 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.6 2021-05-25 07:27:33,545 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.58 2021-05-25 07:27:38,673 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.61 2021-05-25 07:27:43,769 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.89 2021-05-25 07:27:48,880 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 67.03 2021-05-25 07:27:53,980 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.72 2021-05-25 07:27:59,095 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 76.79 2021-05-25 07:28:04,201 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:28:09,302 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:28:19,415 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:28:19,539 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 81.9259998798 sec 2021-05-25 07:28:19,542 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:28:19,763 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:28:20,013 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.936666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.466674444574 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.05511758529 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 66.8255529075 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.780013000217 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 46.6347277401 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 62.3160386006 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.37162713955 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 67.8911315189 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:28:20,036 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:28:25,163 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 2.85 2021-05-25 07:28:30,315 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 5.98 2021-05-25 07:28:35,471 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 9.53 2021-05-25 07:28:40,637 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.89 2021-05-25 07:28:45,788 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 17.02 2021-05-25 07:28:50,937 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 20.9 2021-05-25 07:28:56,085 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 24.36 2021-05-25 07:29:01,219 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 28.39 2021-05-25 07:29:06,378 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-25 07:29:11,519 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:29:16,628 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:29:21,739 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:29:26,822 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:29:31,914 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:29:42,053 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:29:42,204 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 82.1940000057 sec 2021-05-25 07:29:42,207 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:29:42,384 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:29:53,713 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.11 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.60674344572 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.1426952435 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 83.5688810373 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.05455395299 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 42.1995906345 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.295 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 82.9430490508 | Cluster node | | 172.23.104.247 | ['kv'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:29:53,752 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:29:58,857 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 1.91 2021-05-25 07:30:04,038 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 7.41 2021-05-25 07:30:09,180 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 10.2 2021-05-25 07:30:14,367 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 15.48 2021-05-25 07:30:19,513 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 18.55 2021-05-25 07:30:24,650 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 22.86 2021-05-25 07:30:29,775 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.84 2021-05-25 07:30:34,967 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 32.92 2021-05-25 07:30:40,141 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 35.7 2021-05-25 07:30:45,328 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 38.38 2021-05-25 07:30:50,473 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:30:55,575 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:31:00,678 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:31:10,811 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:31:10,865 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 77.1560001373 sec 2021-05-25 07:31:10,867 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:31:11,019 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:31:11,240 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.56695223174 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.496650111663 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 1.58505283509 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 64.585 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.0531648945 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 76.1174627512 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 73.4249675508 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 72.9704504925 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.795 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 62.4233333333 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:31:11,265 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:31:16,391 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 7.97 2021-05-25 07:31:21,561 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 16.1 2021-05-25 07:31:26,724 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 23.44 2021-05-25 07:31:31,888 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 29.61 2021-05-25 07:31:37,053 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 37.26 2021-05-25 07:31:42,194 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:31:47,306 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:31:52,440 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:31:57,543 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:32:02,642 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:32:12,763 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:32:12,917 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 61.6800000668 sec 2021-05-25 07:32:12,917 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:32:13,105 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:32:19,026 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.77485375244 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.36152730788 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.6698055032 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 67.9416666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.31678861314 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 49.5325077915 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 55.1292521542 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.8333333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 49.5551289542 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:32:19,049 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:32:24,176 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.57 2021-05-25 07:32:29,358 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 8.65 2021-05-25 07:32:34,572 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 14.39 2021-05-25 07:32:39,726 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 20.0 2021-05-25 07:32:44,888 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 25.73 2021-05-25 07:32:50,023 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.94 2021-05-25 07:32:55,164 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.56 2021-05-25 07:33:00,301 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.3 2021-05-25 07:33:05,437 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:33:10,596 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:33:15,734 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:33:20,842 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:33:25,933 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:33:36,062 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:33:36,217 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 77.1960000992 sec 2021-05-25 07:33:36,219 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:33:36,404 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:33:36,615 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 14.115 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 15.2597456709 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 21.5913068116 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.7438042699 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 14.7138237941 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 36.4013046228 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 52.37 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 23.1275624146 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 54.8707521541 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:33:36,640 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:33:41,749 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 64.46 2021-05-25 07:33:46,841 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:33:51,924 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:33:57,013 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 83.33 2021-05-25 07:34:07,160 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:34:07,296 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 30.6830000877 sec 2021-05-25 07:34:07,299 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:34:07,470 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:34:19,809 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.92341539026 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 13.5168919482 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 45.444242596 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 51.6916102797 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 16.6397226713 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 63.3322777954 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.03870129 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 59.4688433739 | Cluster node | | 172.23.107.142 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:34:19,844 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:34:24,971 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 3.75 2021-05-25 07:34:30,134 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 9.84 2021-05-25 07:34:35,308 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 15.7 2021-05-25 07:34:40,460 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 21.09 2021-05-25 07:34:45,611 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 27.5 2021-05-25 07:34:50,742 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 34.13 2021-05-25 07:34:55,858 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 39.76 2021-05-25 07:35:00,990 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:35:06,078 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 58.68 2021-05-25 07:35:11,170 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:35:16,263 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:35:21,355 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:35:31,476 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:35:31,532 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 71.7279999256 sec 2021-05-25 07:35:31,535 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:35:31,723 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:35:31,946 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 15.5988532951 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.698368251746 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 4.37188526093 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 47.89079818 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 22.1081405427 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 52.342460959 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 38.3236821086 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 45.545759096 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 3.38661022316 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 48.7229993657 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:35:31,979 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:35:37,134 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 6.25 2021-05-25 07:35:42,322 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.84 2021-05-25 07:35:47,497 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 24.06 2021-05-25 07:35:52,711 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 07:35:57,872 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 39.14 2021-05-25 07:36:03,032 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:36:08,173 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:36:13,302 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:36:18,404 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:36:23,526 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:36:33,651 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:36:33,772 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 61.8280000687 sec 2021-05-25 07:36:33,773 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:36:33,950 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:36:39,859 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.22516125806 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.05666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 9.295 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 48.0198679912 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 24.895 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 41.4066666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 55.9216786018 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.42507375123 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 51.0683333333 | Cluster node | | 172.23.104.247 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:36:39,891 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:36:45,006 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:36:50,095 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:36:55,194 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:37:05,299 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:37:05,368 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 25.5120000839 sec 2021-05-25 07:37:05,369 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:37:05,545 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:37:05,759 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.04671744529 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.45688189606 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 6.3 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 37.0054331522 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.67842261409 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 6.69166666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 12.7364543924 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 41.3530451015 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.53489108515 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 40.365 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:37:05,785 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:37:10,970 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:37:16,052 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:37:21,151 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:37:26,244 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.55 2021-05-25 07:37:31,338 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.55 2021-05-25 07:37:36,424 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.59 2021-05-25 07:37:41,513 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 32.59 2021-05-25 07:37:46,601 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.1 2021-05-25 07:37:51,710 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.66 2021-05-25 07:37:56,822 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 36.21 2021-05-25 07:38:01,934 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 36.21 2021-05-25 07:38:07,017 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 37.24 2021-05-25 07:38:12,108 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 37.76 2021-05-25 07:38:17,207 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 39.31 2021-05-25 07:38:22,306 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.86 2021-05-25 07:38:27,398 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.86 2021-05-25 07:38:32,487 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 41.9 2021-05-25 07:38:37,578 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 42.41 2021-05-25 07:38:42,667 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 07:38:47,750 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:38:52,874 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:38:57,977 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:39:08,115 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:39:08,276 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 122.519999981 sec 2021-05-25 07:39:08,279 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:39:08,483 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:39:14,471 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.19333333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.38666666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 34.869418843 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.12683756125 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 11.3787126238 | --- OUT ---> | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 25.1062482292 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 45.834861162 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.3551892532 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 48.4358072635 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:39:14,500 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:39:19,640 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:39:24,740 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:39:29,839 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:39:34,928 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 32.14 2021-05-25 07:39:40,019 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 34.29 2021-05-25 07:39:45,114 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 34.29 2021-05-25 07:39:50,211 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 07:39:55,312 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 38.57 2021-05-25 07:40:00,408 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 38.57 2021-05-25 07:40:05,499 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 07:40:10,617 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 07:40:15,727 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 07:40:20,813 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 07:40:25,898 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:40:30,990 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:40:36,098 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 07:40:46,240 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:40:46,361 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 91.8940000534 sec 2021-05-25 07:40:46,365 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:40:46,551 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:40:52,809 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.20833333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.0382660289 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 17.9163680605 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 34.8460858986 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.09176819614 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 18.5672855762 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 30.6521775363 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.1833333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 34.0377660372 | Cluster node | | 172.23.104.247 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 07:40:52,846 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:40:57,993 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:41:03,085 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:41:08,214 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:41:13,328 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 64.08 2021-05-25 07:41:18,447 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 71.43 2021-05-25 07:41:23,563 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 71.08 2021-05-25 07:41:28,710 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 70.72 2021-05-25 07:41:33,816 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 78.38 2021-05-25 07:41:38,938 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:41:44,035 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:41:49,124 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:41:54,214 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:41:59,311 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:42:04,404 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:42:09,506 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 76.33 2021-05-25 07:42:14,628 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:42:19,720 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:42:24,809 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:42:29,911 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:42:35,017 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:42:40,105 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:42:50,220 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:42:50,280 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 117.472999811 sec 2021-05-25 07:43:01,153 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 11 #### 2021-05-25 07:43:01,155 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:43:01,344 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.137 2021-05-25 07:43:01,608 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 2.75824139195 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.75842528084 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.18988016866 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 38.0470650978 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.33684456149 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 9.92199739991 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 13.6478784041 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 15.8319305322 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 7.34654422426 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 17.475 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:43:01,638 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:43:06,755 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:43:11,844 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:43:16,953 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:43:22,072 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.69 2021-05-25 07:43:27,203 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 63.67 2021-05-25 07:43:32,322 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.18 2021-05-25 07:43:37,433 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.36 2021-05-25 07:43:42,532 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.31 2021-05-25 07:43:47,641 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.35 2021-05-25 07:43:52,743 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.69 2021-05-25 07:43:57,835 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.75 2021-05-25 07:44:02,928 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.89 2021-05-25 07:44:08,032 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 72.4 2021-05-25 07:44:13,125 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 72.4 2021-05-25 07:44:18,233 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 72.42 2021-05-25 07:44:23,338 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 72.46 2021-05-25 07:44:28,438 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.23 2021-05-25 07:44:33,569 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 78.78 2021-05-25 07:44:38,687 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:44:43,782 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 07:44:53,924 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:44:54,045 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 112.440999985 sec 2021-05-25 07:44:54,048 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:44:54,236 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:44:54,456 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.548333333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.355 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.6299691641 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 57.8376306272 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.17680294672 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 47.8124635411 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 53.3073346333 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.56328122396 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 55.1816666667 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:44:54,487 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:45:00,848 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:45:05,996 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 4.86 2021-05-25 07:45:11,148 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 8.98 2021-05-25 07:45:16,283 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 12.78 2021-05-25 07:45:21,421 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 15.54 2021-05-25 07:45:26,553 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 19.17 2021-05-25 07:45:31,694 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 22.58 2021-05-25 07:45:36,798 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 25.92 2021-05-25 07:45:41,984 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 29.06 2021-05-25 07:45:47,124 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 32.39 2021-05-25 07:45:52,263 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:45:57,371 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 07:46:02,523 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:46:07,632 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:46:12,729 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:46:22,855 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:46:23,003 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 88.5490000248 sec 2021-05-25 07:46:23,005 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:46:23,183 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:46:33,579 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.8428104793 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.25850861695 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 10.3603453448 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 83.1502808287 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.2071770255 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 57.705 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.29666666667 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 82.574948254 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:46:33,608 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:46:38,730 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.37 2021-05-25 07:46:43,979 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 5.44 2021-05-25 07:46:49,150 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.75 2021-05-25 07:46:54,308 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 11.09 2021-05-25 07:46:59,448 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.95 2021-05-25 07:47:04,621 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 17.84 2021-05-25 07:47:09,775 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 23.27 2021-05-25 07:47:14,905 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 28.97 2021-05-25 07:47:20,035 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 34.81 2021-05-25 07:47:25,190 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 38.66 2021-05-25 07:47:30,365 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:47:35,460 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:47:40,598 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:47:50,729 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:47:50,786 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 77.2109999657 sec 2021-05-25 07:47:50,789 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:47:50,967 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:47:51,187 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 13.4871162372 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.755 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 2.13666666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 59.3493224887 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.25486241896 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 69.8188363527 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 68.4372812427 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 76.3284218948 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.21662972284 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 66.8488858519 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:47:51,213 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:47:56,341 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 7.58 2021-05-25 07:48:01,522 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 16.87 2021-05-25 07:48:06,677 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 26.56 2021-05-25 07:48:11,868 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 36.64 2021-05-25 07:48:17,029 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:48:22,164 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:48:27,283 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:48:32,375 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:48:37,464 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:48:47,584 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:48:47,723 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 56.5390000343 sec 2021-05-25 07:48:47,723 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:48:47,894 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:48:53,470 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.87333333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.63666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 13.4145528482 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.16321394643 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 50.1291774315 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 84.2461415381 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 78.8286861886 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 14.7152452541 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 80.0073221061 | --- OUT ---> | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:48:53,497 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:48:58,608 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 1.99 2021-05-25 07:49:03,813 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 7.86 2021-05-25 07:49:09,013 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 13.61 2021-05-25 07:49:14,249 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 19.41 2021-05-25 07:49:19,467 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 23.46 2021-05-25 07:49:24,622 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 28.33 2021-05-25 07:49:29,779 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.61 2021-05-25 07:49:34,977 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 38.53 2021-05-25 07:49:40,117 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:49:45,239 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:49:50,398 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:49:55,484 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:50:00,575 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:50:10,690 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:50:10,805 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 77.3389999866 sec 2021-05-25 07:50:10,808 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:50:10,982 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:50:11,221 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.858333333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.495008250138 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 1.53833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.9276024133 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.678333333333 | --- OUT ---> | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 58.9843164053 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 65.9878004067 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 53.231558948 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.15673855795 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:50:11,244 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:50:16,351 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 64.46 2021-05-25 07:50:21,499 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:50:26,582 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 07:50:31,658 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 77.78 2021-05-25 07:50:41,790 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:50:41,938 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 30.7189998627 sec 2021-05-25 07:50:41,941 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:50:42,121 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:50:52,355 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 47.9408676522 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.81166666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.85 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 16.535551185 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 17.8666666667 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 15.13 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 27.0539211601 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.65314489517 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.107.131 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:50:52,381 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:50:57,483 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 1.64 2021-05-25 07:51:02,598 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 10.93 2021-05-25 07:51:07,720 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 20.6 2021-05-25 07:51:12,868 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.29 2021-05-25 07:51:17,967 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 29.07 2021-05-25 07:51:23,084 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 33.19 2021-05-25 07:51:28,188 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 36.96 2021-05-25 07:51:33,306 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:51:38,404 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 07:51:43,507 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 07:51:48,608 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:51:53,723 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:52:03,894 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:52:03,970 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 71.618999958 sec 2021-05-25 07:52:03,973 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:52:13,217 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:52:13,552 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.835 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.43666666667 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.13511891865 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 60.393019651 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.935046752338 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 48.3583333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 17.1586193103 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 30.0528324528 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.5114748088 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 29.1388189803 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 07:52:13,595 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:52:18,734 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 1.41 2021-05-25 07:52:23,948 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 10.08 2021-05-25 07:52:29,148 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 15.46 2021-05-25 07:52:34,634 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 20.7 2021-05-25 07:52:39,802 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 27.89 2021-05-25 07:52:44,960 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 34.37 2021-05-25 07:52:50,157 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 39.61 2021-05-25 07:52:55,335 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:53:00,479 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 07:53:05,611 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:53:10,759 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:53:15,890 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 07:53:26,056 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:53:26,194 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 72.6459999084 sec 2021-05-25 07:53:26,196 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:53:26,447 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:53:32,424 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 11.9268654478 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.98011633527 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 29.1154852581 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 21.0666666667 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 54.9820549203 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 66.8988850186 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 79.8966666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.7283333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 66.7710345746 | Cluster node | | 172.23.107.129 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:53:32,476 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:53:37,615 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:53:42,737 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:53:47,849 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:53:57,990 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:53:58,062 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 25.6440000534 sec 2021-05-25 07:53:58,065 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:53:58,321 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:53:58,595 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.48144259454 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.61845394846 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.18654689089 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 10.805540277 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.69821005966 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 21.2216666667 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 55.2101736725 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 39.2966666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 8.35986066899 | --- OUT ---> | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 35.8839313989 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:53:58,630 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:54:03,753 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:54:08,858 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:54:13,969 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 07:54:19,056 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 07:54:24,161 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 07:54:29,265 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 07:54:34,351 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 07:54:39,447 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 07:54:44,552 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 07:54:49,648 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 07:54:54,736 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 07:54:59,825 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 07:55:04,927 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 07:55:10,035 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 07:55:15,127 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 42.5 2021-05-25 07:55:20,217 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:55:25,315 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:55:30,401 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:55:40,510 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:55:40,631 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 102.038000107 sec 2021-05-25 07:55:40,632 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 07:55:40,857 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:55:47,242 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.82666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.52658244725 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 18.6483333333 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 4.76658722355 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.08993183447 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 15.4966666667 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 14.6621554052 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 18.8033333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 22.2394453056 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 07:55:47,266 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:55:52,368 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 07:55:57,454 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 07:56:02,546 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 32.14 2021-05-25 07:56:07,632 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 32.14 2021-05-25 07:56:12,752 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 34.29 2021-05-25 07:56:17,845 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 34.29 2021-05-25 07:56:22,948 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 07:56:28,048 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 07:56:33,181 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.57 2021-05-25 07:56:38,259 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 07:56:43,342 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 07:56:48,428 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.71 2021-05-25 07:56:53,545 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 07:56:58,650 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:57:03,744 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:57:08,828 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 07:57:18,934 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:57:19,059 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 91.8210000992 sec 2021-05-25 07:57:19,061 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 07:57:19,289 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:57:24,648 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.76675500442 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.29162847286 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.02550085 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.25 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 13.1807879475 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 17.8130364494 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 17.0313828103 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.51685056169 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 4.52465345381 | Cluster node | | 172.23.107.129 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 07:57:24,687 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:57:29,828 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:57:34,921 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:57:40,015 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:57:45,111 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 62.74 2021-05-25 07:57:50,246 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 70.65 2021-05-25 07:57:55,351 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 76.69 2021-05-25 07:58:00,443 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:58:05,528 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 07:58:10,611 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:58:15,697 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 76.71 2021-05-25 07:58:20,788 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 07:58:25,871 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:58:30,993 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:58:36,088 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:58:41,173 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 07:58:46,262 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:58:51,344 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 77.96 2021-05-25 07:58:56,427 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 07:59:06,546 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 07:59:06,601 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 101.953999996 sec 2021-05-25 07:59:18,650 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 12 #### 2021-05-25 07:59:18,651 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 07:59:18,851 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 07:59:19,085 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.49845830764 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.49679150624 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 9.61016016934 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 15.39 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.13175219587 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 20.2583333333 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 38.5410243163 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 36.665 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.1982300295 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 45.5810160917 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 07:59:19,114 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 07:59:24,226 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:59:29,368 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:59:34,529 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 07:59:39,714 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 61.87 2021-05-25 07:59:44,859 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.53 2021-05-25 07:59:50,028 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.91 2021-05-25 07:59:55,285 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.88 2021-05-25 08:00:00,470 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.89 2021-05-25 08:00:05,671 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 66.16 2021-05-25 08:00:10,822 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 68.08 2021-05-25 08:00:15,961 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 68.72 2021-05-25 08:00:21,108 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 68.8 2021-05-25 08:00:26,219 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 73.89 2021-05-25 08:00:31,380 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:00:36,490 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:00:46,618 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:00:47,052 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 87.9709999561 sec 2021-05-25 08:00:47,055 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:00:47,259 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:00:47,476 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.57358578619 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.48988367054 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.64161618851 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 58.645 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.2205610281 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 41.6910785619 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 53.6619811916 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.8785646427 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 46.6194065865 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:00:47,503 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:00:52,607 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:00:58,151 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.27 2021-05-25 08:01:03,381 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 4.83 2021-05-25 08:01:08,630 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 8.87 2021-05-25 08:01:13,815 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 13.75 2021-05-25 08:01:19,036 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 18.05 2021-05-25 08:01:24,243 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 22.04 2021-05-25 08:01:29,430 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 26.64 2021-05-25 08:01:34,621 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 30.99 2021-05-25 08:01:39,844 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:01:44,961 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:01:50,095 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:01:55,220 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:02:00,338 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:02:10,506 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:02:10,650 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 83.1749999523 sec 2021-05-25 08:02:10,651 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:02:10,835 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:02:22,051 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 2.29496175064 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.85179505984 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 3.96146859324 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 58.2019400647 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.8 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 74.1670416854 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 76.1732551163 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.45340755679 | Cluster node | | 172.23.107.131 | ['kv'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:02:22,108 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:02:27,226 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 1.65 2021-05-25 08:02:32,437 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 6.57 2021-05-25 08:02:37,713 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 9.71 2021-05-25 08:02:42,890 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 16.13 2021-05-25 08:02:48,069 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 18.48 2021-05-25 08:02:53,303 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 22.44 2021-05-25 08:02:58,592 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 27.96 2021-05-25 08:03:03,762 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 33.77 2021-05-25 08:03:08,973 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 36.93 2021-05-25 08:03:14,128 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:03:19,230 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:03:24,326 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:03:34,507 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:03:34,565 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 72.5169999599 sec 2021-05-25 08:03:34,568 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:03:34,752 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:03:34,996 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 16.5866666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 10.5498241696 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 18.7023982134 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 68.176060798 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.37651039149 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 49.2599082186 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 65.4405759904 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 67.1605720191 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.9383333333 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 67.11 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:03:35,033 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:03:40,457 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 8.75 2021-05-25 08:03:45,694 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 17.73 2021-05-25 08:03:51,029 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 26.01 2021-05-25 08:03:56,263 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 35.39 2021-05-25 08:04:01,428 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:04:06,571 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:04:11,677 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:04:16,773 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:04:21,881 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:04:32,048 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:04:32,414 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 57.4210000038 sec 2021-05-25 08:04:32,415 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:04:32,598 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:04:38,977 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 10.6264895585 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 29.6345060916 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 17.0074829592 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 51.2225611281 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.4964917251 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 79.2690243659 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 58.1513949535 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 54.0624322928 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.5744808506 | Cluster node | | 172.23.107.131 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:04:39,075 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:04:44,213 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.35 2021-05-25 08:04:49,418 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 6.98 2021-05-25 08:04:54,674 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 12.2 2021-05-25 08:04:59,875 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 16.6 2021-05-25 08:05:05,119 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 21.93 2021-05-25 08:05:10,348 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 26.51 2021-05-25 08:05:15,505 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.14 2021-05-25 08:05:20,687 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.78 2021-05-25 08:05:25,842 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 37.6 2021-05-25 08:05:30,990 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:05:36,138 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:05:41,273 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:05:46,397 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:05:51,529 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:06:01,746 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:06:02,828 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 83.8550000191 sec 2021-05-25 08:06:02,831 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:06:03,017 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:06:03,302 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 10.0238345251 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.54351811727 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 20.419319356 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 51.5365642291 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.16818053032 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 70.9439946582 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 45.3418219393 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 21.9529674505 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 50.6774661267 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:06:03,345 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:06:08,490 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:06:13,607 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-25 08:06:18,749 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:06:23,865 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:06:28,963 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:06:39,089 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:06:39,237 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 35.9389998913 sec 2021-05-25 08:06:39,240 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:06:39,441 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:06:50,717 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.76997050049 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.9466666667 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 18.4110529649 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 49.6858280971 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 49.9216692777 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 28.3090563648 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.28657855702 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 49.4550181661 | Cluster node | | 172.23.107.142 | ['cbas'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:06:50,743 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:06:55,936 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 3.47 2021-05-25 08:07:01,135 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 12.31 2021-05-25 08:07:06,296 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 17.19 2021-05-25 08:07:11,438 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 22.79 2021-05-25 08:07:16,566 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 27.23 2021-05-25 08:07:21,719 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 30.5 2021-05-25 08:07:26,826 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 33.88 2021-05-25 08:07:31,930 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 35.8 2021-05-25 08:07:37,078 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 38.49 2021-05-25 08:07:42,203 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:07:47,311 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:07:52,411 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 08:07:57,493 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:08:02,584 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:08:12,701 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:08:12,757 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 82.0449998379 sec 2021-05-25 08:08:12,780 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:08:12,934 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:08:13,138 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 48.7233758875 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.55849264154 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 7.03 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 25.8604310072 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.67371158077 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 46.9551014966 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 51.2513347571 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 66.5368678548 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.82 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 26.5075502517 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:08:13,164 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:08:18,303 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 8.05 2021-05-25 08:08:23,517 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 18.2 2021-05-25 08:08:28,819 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 27.27 2021-05-25 08:08:34,013 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 36.64 2021-05-25 08:08:39,237 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:08:44,361 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:08:49,486 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:08:54,585 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:08:59,683 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:09:09,855 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:09:09,974 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 56.8399999142 sec 2021-05-25 08:09:09,976 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:09:10,178 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:09:15,589 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 9.06166666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.74336238937 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 4.4518150605 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 53.9332022266 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.24 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 55.7540707655 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 56.3014566181 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 68.5689522984 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.31324477925 | Cluster node | | 172.23.107.131 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:09:15,631 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:09:20,759 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:09:25,848 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:09:30,934 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:09:41,056 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:09:41,125 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 25.5409998894 sec 2021-05-25 08:09:41,128 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:09:41,351 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:09:41,622 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.653322444626 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.469968668755 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 2.63162280629 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 14.8924112872 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.496674944582 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 42.5812042731 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 26.9036548173 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 27.8654644244 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.42004033401 | --- OUT ---> | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.97983440171 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 08:09:41,660 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:09:46,773 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:09:51,875 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:09:56,979 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:10:02,076 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:07,187 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:12,276 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:17,358 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:22,440 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:27,536 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:32,624 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 32.31 2021-05-25 08:10:37,730 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.46 2021-05-25 08:10:42,844 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.46 2021-05-25 08:10:47,940 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 08:10:53,026 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 08:10:58,112 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 38.08 2021-05-25 08:11:03,230 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.38 2021-05-25 08:11:08,325 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 41.54 2021-05-25 08:11:13,414 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 08:11:18,540 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:11:23,625 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:11:28,709 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:11:38,818 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:11:38,950 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 117.330000162 sec 2021-05-25 08:11:38,953 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:11:39,155 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:11:45,144 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.68507808463 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.82673044551 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 6.89075630252 | --- OUT ---> | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 18.8003133386 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 16.5866666667 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 44.9742504292 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 20.9113181447 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 21.7963033949 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 7.79505582137 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:11:45,176 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:11:50,286 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:11:55,375 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:12:00,496 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:12:05,578 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:12:10,678 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:12:15,765 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 32.81 2021-05-25 08:12:20,846 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 34.22 2021-05-25 08:12:25,956 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 34.22 2021-05-25 08:12:31,059 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 08:12:36,154 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 08:12:41,240 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.09 2021-05-25 08:12:46,332 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.03 2021-05-25 08:12:51,424 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.03 2021-05-25 08:12:56,523 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 38.44 2021-05-25 08:13:01,611 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 08:13:06,740 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.84 2021-05-25 08:13:11,838 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 08:13:16,921 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 08:13:22,012 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 42.66 2021-05-25 08:13:27,118 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 44.06 2021-05-25 08:13:32,240 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:13:37,332 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:13:42,450 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:13:52,572 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:13:52,688 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 127.55099988 sec 2021-05-25 08:13:52,690 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:13:52,865 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:13:58,740 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.30847180786 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.86176205874 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 21.1966666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.45340755679 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 44.840919318 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 39.2193463442 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 27.032117202 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.4022134071 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.88804054279 | Cluster node | | 172.23.107.127 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 08:13:58,786 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:14:03,928 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:14:09,040 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:14:14,178 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:14:19,306 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 65.55 2021-05-25 08:14:24,476 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 73.83 2021-05-25 08:14:29,608 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 75.81 2021-05-25 08:14:34,730 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 73.4 2021-05-25 08:14:39,852 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 74.58 2021-05-25 08:14:44,970 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 08:14:50,065 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 08:14:55,164 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:15:00,260 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 78.39 2021-05-25 08:15:05,367 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:15:10,453 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:15:15,545 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:15:20,641 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 78.13 2021-05-25 08:15:25,750 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:15:30,855 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:15:35,943 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 75.36 2021-05-25 08:15:41,046 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:15:46,151 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:15:56,289 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:15:56,365 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 117.628000021 sec 2021-05-25 08:16:07,219 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 13 #### 2021-05-25 08:16:07,221 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:16:07,436 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:16:07,644 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.70480983967 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.23674122471 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 13.826897115 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 13.3821127371 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.06676777946 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 22.3687061451 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 28.1516666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 23.9429096069 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 8.02026734224 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 12.7629079031 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:16:07,680 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:16:12,835 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:16:17,941 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:16:23,092 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:16:28,250 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.71 2021-05-25 08:16:33,427 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 62.22 2021-05-25 08:16:38,571 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 63.61 2021-05-25 08:16:43,769 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 64.72 2021-05-25 08:16:48,961 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 65.4 2021-05-25 08:16:54,092 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.71 2021-05-25 08:16:59,236 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 67.82 2021-05-25 08:17:04,420 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.29 2021-05-25 08:17:09,569 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.5 2021-05-25 08:17:14,733 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.53 2021-05-25 08:17:19,877 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.54 2021-05-25 08:17:25,019 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.53 2021-05-25 08:17:30,155 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.52 2021-05-25 08:17:35,263 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.53 2021-05-25 08:17:40,378 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.54 2021-05-25 08:17:45,489 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.76 2021-05-25 08:17:50,630 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.06 2021-05-25 08:17:55,763 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 75.42 2021-05-25 08:18:00,897 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.43 2021-05-25 08:18:06,045 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:18:11,171 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:18:16,280 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:18:21,428 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:18:31,565 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:18:31,703 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 144.063999891 sec 2021-05-25 08:18:31,707 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:18:31,914 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:18:32,173 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.35972134262 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.385 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 44.7116666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.20327994533 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 41.4563818794 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 47.106348173 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 32.0066666667 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.62823952934 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.75657072382 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:18:32,221 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:18:37,401 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 3.75 2021-05-25 08:18:42,556 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 8.61 2021-05-25 08:18:47,805 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 12.07 2021-05-25 08:18:53,012 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 15.51 2021-05-25 08:18:58,233 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.16 2021-05-25 08:19:03,453 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 22.41 2021-05-25 08:19:08,651 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 25.54 2021-05-25 08:19:13,875 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 28.91 2021-05-25 08:19:19,131 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 31.67 2021-05-25 08:19:24,328 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:19:29,480 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:19:34,611 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-25 08:19:39,720 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:19:44,815 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:19:54,946 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:19:55,078 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 82.9079999924 sec 2021-05-25 08:19:55,081 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:19:55,270 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:20:07,352 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.84266577918 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.37348955816 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 44.5038918613 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.67442170522 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 87.7056147193 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 85.2088263725 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.845 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 19.7130047833 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:20:07,401 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:20:12,601 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.65 2021-05-25 08:20:17,904 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 4.94 2021-05-25 08:20:23,155 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 7.59 2021-05-25 08:20:28,344 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 12.0 2021-05-25 08:20:33,588 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 17.81 2021-05-25 08:20:38,872 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 21.64 2021-05-25 08:20:44,117 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 28.09 2021-05-25 08:20:49,354 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.54 2021-05-25 08:20:54,549 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 37.04 2021-05-25 08:20:59,701 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:21:04,805 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:21:09,897 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:21:20,069 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:21:20,157 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 72.8090000153 sec 2021-05-25 08:21:20,158 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:21:20,382 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:21:20,650 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.96480117329 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.288323722543 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 81.9003016616 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 69.5897760597 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.80077453386 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 65.1844197403 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 61.876270791 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 61.7723037949 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.95166666667 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.74497091715 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 08:21:20,690 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:21:25,858 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 7.34 2021-05-25 08:21:31,089 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 16.33 2021-05-25 08:21:36,339 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 25.54 2021-05-25 08:21:41,517 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.61 2021-05-25 08:21:46,697 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:21:51,829 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:21:56,948 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:22:02,056 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:22:07,157 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:22:17,299 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:22:17,923 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 57.2760000229 sec 2021-05-25 08:22:17,924 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:22:18,197 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:22:25,625 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.87666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.391660138998 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 49.6099869996 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.03163280612 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 58.2263924536 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 58.6202873429 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 64.6321544051 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 3.72481375931 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 19.7558046437 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 08:22:25,694 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:22:30,861 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 1.94 2021-05-25 08:22:36,092 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 7.92 2021-05-25 08:22:41,259 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 13.31 2021-05-25 08:22:46,433 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 18.65 2021-05-25 08:22:51,611 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 22.52 2021-05-25 08:22:56,773 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 26.86 2021-05-25 08:23:01,907 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.73 2021-05-25 08:23:07,101 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.77 2021-05-25 08:23:12,256 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:23:17,365 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:23:22,476 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:23:27,585 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:23:32,681 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:23:37,805 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:23:47,959 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:23:48,096 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 82.4769999981 sec 2021-05-25 08:23:48,099 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:23:48,322 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:23:48,558 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.7404687187 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.37340622344 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 68.4378072968 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 73.2134808954 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.57481417286 | --- OUT ---> | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 70.9652149274 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 59.455 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.65 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 34.352100686 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:23:48,595 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:23:53,707 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 56.83 2021-05-25 08:23:58,792 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:24:03,872 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:24:08,960 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:24:19,085 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:24:19,203 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 30.6480000019 sec 2021-05-25 08:24:19,204 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:24:19,374 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:24:31,138 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 48.6337432104 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.47818406053 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 20.1313311445 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 59.5936569213 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 38.1516666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 44.5581519384 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 12.7097881702 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 30.1888676263 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:24:31,164 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:24:36,286 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 4.98 2021-05-25 08:24:41,418 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 16.05 2021-05-25 08:24:46,569 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 21.9 2021-05-25 08:24:51,723 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 25.44 2021-05-25 08:24:56,869 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 28.72 2021-05-25 08:25:01,992 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 32.37 2021-05-25 08:25:07,092 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 36.13 2021-05-25 08:25:12,201 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:25:17,311 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:25:22,398 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 08:25:27,500 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:25:32,615 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:25:37,746 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:25:47,898 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:25:47,993 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 76.861000061 sec 2021-05-25 08:25:47,996 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:25:48,219 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:25:48,490 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.78666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.38189212614 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 68.0478007967 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 67.2141819821 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.27485750475 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 64.0372660456 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 75.3637439376 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 61.6916666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.9088787773 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 11.4568576143 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:25:48,530 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:25:53,696 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 6.17 2021-05-25 08:25:58,887 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 15.39 2021-05-25 08:26:04,036 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 24.92 2021-05-25 08:26:09,197 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.98 2021-05-25 08:26:14,384 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:26:19,503 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:26:24,631 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:26:29,762 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:26:34,868 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:26:45,020 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:26:45,809 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 57.3199999332 sec 2021-05-25 08:26:45,809 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:26:46,058 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:26:53,180 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 15.0530824486 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.99849995 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 57.8380720643 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 62.5873367907 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.78820352994 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 59.9223346278 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 57.5002500083 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.6847052549 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 17.3724647101 | Cluster node | | 172.23.104.247 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:26:53,221 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:26:58,367 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 08:27:03,460 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:27:08,546 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:27:18,657 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:27:18,714 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 25.5399999619 sec 2021-05-25 08:27:18,717 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:27:18,884 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:27:19,111 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.808306389787 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.338338972316 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 14.7135785596 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 44.5816666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.65 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 1.04003466782 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 45.9107651794 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 42.4997499917 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 3.07171786196 | --- OUT ---> | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.95169919499 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 08:27:19,137 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:27:24,351 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:27:29,441 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:27:34,539 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:27:39,645 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:27:44,726 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 32.05 2021-05-25 08:27:49,816 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 32.05 2021-05-25 08:27:54,908 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 08:27:59,996 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 34.09 2021-05-25 08:28:05,092 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 36.14 2021-05-25 08:28:10,207 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 08:28:15,296 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 08:28:20,384 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 38.18 2021-05-25 08:28:25,477 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.23 2021-05-25 08:28:30,576 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 08:28:35,664 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 08:28:40,766 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.27 2021-05-25 08:28:45,855 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:28:50,956 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:28:56,039 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:29:06,145 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:29:06,282 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 107.17200017 sec 2021-05-25 08:29:06,285 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:29:06,479 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:29:12,542 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.25639145362 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.48344139069 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 16.7433333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 33.2048007745 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.30322827953 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 13.5178827372 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 19.2706423547 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 40.7312967685 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.6783333333 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:29:12,565 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:29:17,703 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:29:22,790 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:29:27,882 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:29:32,993 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 08:29:38,081 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 08:29:43,160 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 08:29:48,253 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 08:29:53,364 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 08:29:58,461 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 08:30:03,546 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 08:30:08,631 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 08:30:13,720 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 08:30:18,825 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 08:30:23,914 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-05-25 08:30:29,009 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:30:34,111 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:30:39,193 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:30:49,305 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:30:49,459 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 96.9210000038 sec 2021-05-25 08:30:49,461 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:30:49,657 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:30:55,575 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.59244174401 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.19670655689 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 11.2690846056 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 6.63833333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.35168919482 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 5.52148261725 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 4.19166666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.34992750121 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.22470713525 | Cluster node | | 172.23.104.247 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 08:30:55,628 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:31:00,753 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:31:05,845 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:31:10,953 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:31:16,065 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 64.64 2021-05-25 08:31:21,184 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 68.23 2021-05-25 08:31:26,302 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 69.54 2021-05-25 08:31:31,440 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 72.17 2021-05-25 08:31:36,559 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.88 2021-05-25 08:31:41,645 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.92 2021-05-25 08:31:46,736 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 76.17 2021-05-25 08:31:51,835 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 77.94 2021-05-25 08:31:56,924 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:32:02,029 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:32:07,109 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:32:12,200 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:32:17,282 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:32:22,369 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:32:27,474 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 76.72 2021-05-25 08:32:32,599 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:32:37,687 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:32:47,799 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:32:47,872 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 112.299999952 sec 2021-05-25 08:32:58,408 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 14 #### 2021-05-25 08:32:58,411 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:32:58,635 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:32:58,868 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.963317278045 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.259995666739 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 16.5363910601 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.5639427324 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.65009416824 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 10.8053601787 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 47.2115737191 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 26.0666666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.38164363927 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 1.17166666667 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 08:32:58,901 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:33:04,020 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:33:09,114 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:33:14,233 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:33:19,332 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.85 2021-05-25 08:33:24,470 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 63.45 2021-05-25 08:33:29,568 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.19 2021-05-25 08:33:34,697 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.51 2021-05-25 08:33:39,792 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.17 2021-05-25 08:33:44,917 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.2 2021-05-25 08:33:50,023 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.19 2021-05-25 08:33:55,130 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.18 2021-05-25 08:34:00,221 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.21 2021-05-25 08:34:05,622 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.7 2021-05-25 08:34:10,786 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.01 2021-05-25 08:34:15,917 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.07 2021-05-25 08:34:21,026 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.56 2021-05-25 08:34:26,131 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 73.65 2021-05-25 08:34:31,246 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:34:36,348 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:34:46,480 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:34:46,604 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 107.74000001 sec 2021-05-25 08:34:46,605 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:34:46,776 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 08:34:46,979 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 12.1354045135 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.39008983483 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 81.562473917 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.27666666667 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 53.15 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 80.6236562724 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 77.5803736604 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.0636510609 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 15.7012653157 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:34:47,022 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:34:52,154 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 4.68 2021-05-25 08:34:57,289 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 8.99 2021-05-25 08:35:02,440 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 12.9 2021-05-25 08:35:07,575 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 17.13 2021-05-25 08:35:12,719 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.57 2021-05-25 08:35:17,884 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 24.67 2021-05-25 08:35:23,065 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 27.82 2021-05-25 08:35:28,187 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 31.57 2021-05-25 08:35:33,313 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:35:38,428 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:35:43,540 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:35:48,628 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:35:53,736 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:36:03,848 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:36:03,967 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 76.9900000095 sec 2021-05-25 08:36:03,970 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:36:04,154 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:36:15,592 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.94486758554 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.41972901355 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 58.3887472293 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.1566666667 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 50.444159264 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 42.4083072622 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 21.978699645 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 19.9216666667 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:36:15,619 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:36:21,480 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:36:26,719 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.61 2021-05-25 08:36:31,999 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 1.45 2021-05-25 08:36:37,171 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 4.35 2021-05-25 08:36:42,306 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 7.14 2021-05-25 08:36:47,427 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 7.78 2021-05-25 08:36:52,579 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 14.92 2021-05-25 08:36:57,756 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 16.0 2021-05-25 08:37:02,970 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 18.79 2021-05-25 08:37:08,140 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 27.86 2021-05-25 08:37:13,265 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 34.25 2021-05-25 08:37:18,423 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 37.04 2021-05-25 08:37:23,589 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:37:28,688 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:37:33,789 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:37:43,951 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:37:44,025 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 88.4390001297 sec 2021-05-25 08:37:44,028 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:37:44,194 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:37:44,447 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 19.6404426862 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.08321527975 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 79.3223559215 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 74.7646832628 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 13.6271977335 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 72.1862031034 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 54.7118237275 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 82.0555193884 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.05 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.64347739129 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:37:44,484 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:37:49,621 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 4.69 2021-05-25 08:37:54,825 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 14.84 2021-05-25 08:37:59,969 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 23.05 2021-05-25 08:38:05,134 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.64 2021-05-25 08:38:10,285 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.92 2021-05-25 08:38:15,443 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:38:20,568 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:38:25,684 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:38:30,795 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:38:35,907 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 08:38:46,073 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:38:46,184 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 61.7419998646 sec 2021-05-25 08:38:46,187 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:38:46,388 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:38:51,713 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.90178169636 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.61007683461 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.2778546309 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.76333333333 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 41.1980399347 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 63.67 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 73.2443586501 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.3796206793 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:38:51,737 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:38:56,874 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 1.88 2021-05-25 08:39:02,023 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 7.16 2021-05-25 08:39:07,191 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 13.37 2021-05-25 08:39:12,342 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 19.0 2021-05-25 08:39:17,536 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 22.46 2021-05-25 08:39:22,767 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 25.57 2021-05-25 08:39:27,971 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.38 2021-05-25 08:39:33,111 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.13 2021-05-25 08:39:38,275 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 38.83 2021-05-25 08:39:43,441 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:39:48,578 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:39:53,694 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:39:58,786 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:40:03,905 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:40:14,127 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:40:14,267 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 82.5579998493 sec 2021-05-25 08:40:14,269 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:40:14,499 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:40:14,933 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.8269608987 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.38833333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 68.2806093536 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 57.6026267104 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.52654122431 | --- OUT ---> | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 62.6810446841 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 32.1111314811 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.4966666667 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 7.3867607415 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:40:14,973 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:40:20,117 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 45.55 2021-05-25 08:40:25,232 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:40:30,315 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:40:35,392 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:40:45,509 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:40:45,628 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 30.6979999542 sec 2021-05-25 08:40:45,630 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:40:45,832 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:40:57,078 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.1982300295 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.6967282788 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 51.7215518965 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 53.3066666667 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 16.719721338 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 45.9609006517 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 12.4218736979 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.0058758722 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:40:57,108 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:41:02,233 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 1.22 2021-05-25 08:41:07,390 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 13.03 2021-05-25 08:41:12,506 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 20.49 2021-05-25 08:41:17,621 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 24.08 2021-05-25 08:41:22,746 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 26.58 2021-05-25 08:41:27,875 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 30.58 2021-05-25 08:41:33,013 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 34.46 2021-05-25 08:41:38,153 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 38.12 2021-05-25 08:41:43,259 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:41:48,351 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:41:53,441 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:41:58,548 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:42:03,660 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:42:13,822 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:42:13,875 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 76.8020000458 sec 2021-05-25 08:42:13,878 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:42:14,029 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:42:14,230 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.47032115911 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.89328511191 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 45.3383333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 67.6817092652 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.7249379177 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 57.3185772859 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 35.8066666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 49.1933602213 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.41666666667 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.1204060203 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:42:14,273 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:42:19,427 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 3.67 2021-05-25 08:42:24,596 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 11.02 2021-05-25 08:42:29,759 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 20.16 2021-05-25 08:42:34,917 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 28.28 2021-05-25 08:42:40,069 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 35.86 2021-05-25 08:42:45,232 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:42:50,348 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:42:55,460 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:43:00,595 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:43:05,687 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:43:15,816 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:43:15,953 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 61.7260000706 sec 2021-05-25 08:43:15,954 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:43:16,151 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:43:22,345 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.64125605312 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.88 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 64.9733333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 70.6778446307 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.15816139462 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 54.2984766159 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 54.2983333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.4902248371 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.5346356723 | Cluster node | | 172.23.105.1 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:43:22,369 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:43:27,503 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:43:32,585 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:43:37,684 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:43:47,801 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:43:47,855 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 25.5169999599 sec 2021-05-25 08:43:47,858 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:43:48,055 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:43:48,260 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.54166666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.78828686189 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 63.7506041566 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.1075666051 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.125 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 24.2712621456 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 56.9264357855 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 7.43485736224 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 7.44679077985 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.26718586191 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:43:48,285 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:43:53,417 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:43:58,503 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:44:03,611 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:44:08,697 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:44:13,795 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 08:44:18,885 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 08:44:24,000 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 08:44:29,108 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 08:44:34,223 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 08:44:39,338 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 08:44:44,446 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 08:44:49,561 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 08:44:54,655 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 08:44:59,749 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 08:45:04,864 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 43.5 2021-05-25 08:45:10,016 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:45:15,105 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:45:20,227 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:45:30,345 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:45:30,474 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 102.215999842 sec 2021-05-25 08:45:30,476 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:45:30,645 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:45:35,940 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.94868076737 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.40510675178 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 62.0056332394 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 59.9593969448 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.63822269629 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 30.2106596447 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 49.3225112915 | Cluster node | | 172.23.105.1 | index, n1ql | 7.0.0-5219-enterprise | 12.4702078368 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 8.12333333333 | Cluster node | | 172.23.107.131 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 08:45:35,969 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:45:41,124 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 08:45:46,224 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:45:51,312 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 08:45:56,407 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 33.21 2021-05-25 08:46:01,499 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 08:46:06,588 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 08:46:11,694 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 38.57 2021-05-25 08:46:16,786 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 39.64 2021-05-25 08:46:21,881 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 08:46:26,970 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 42.86 2021-05-25 08:46:32,055 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 08:46:37,137 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:46:42,250 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:46:47,336 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 08:46:57,502 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:46:57,625 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 81.6909999847 sec 2021-05-25 08:46:57,628 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:46:57,829 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:47:03,437 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.20684022801 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.89161847303 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 14.7664205597 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 15.7292152034 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.29826169564 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 13.8814353094 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 19.3260224659 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 8.82803906536 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 9.74284619102 | Cluster node | | 172.23.105.1 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 08:47:03,463 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:47:08,571 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:47:13,661 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:47:18,766 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:47:23,881 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.94 2021-05-25 08:47:28,979 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.49 2021-05-25 08:47:34,065 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 67.99 2021-05-25 08:47:39,167 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 70.41 2021-05-25 08:47:44,322 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 71.99 2021-05-25 08:47:49,436 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 73.56 2021-05-25 08:47:54,549 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.73 2021-05-25 08:47:59,657 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 08:48:04,759 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:48:09,854 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:48:14,953 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:48:20,059 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 08:48:25,164 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:48:30,250 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 76.21 2021-05-25 08:48:35,348 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:48:40,437 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:48:45,533 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:48:50,638 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.17 2021-05-25 08:48:55,760 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:49:00,848 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:49:10,957 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:49:11,016 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 127.579999924 sec 2021-05-25 08:49:22,220 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 15 #### 2021-05-25 08:49:22,223 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:49:22,408 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:49:22,628 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.97685485615 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.55333333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 47.5784140529 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.7074548758 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.75152080799 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 15.2572363369 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 21.2 | Cluster node | | 172.23.105.1 | eventing | 7.0.0-5219-enterprise | 10.8083333333 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.04343405723 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 6.145 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:49:22,651 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:49:27,755 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:49:32,858 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:49:37,950 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 08:49:43,053 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 61.03 2021-05-25 08:49:48,148 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 63.92 2021-05-25 08:49:53,259 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 66.79 2021-05-25 08:49:58,365 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 67.7 2021-05-25 08:50:03,513 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.09 2021-05-25 08:50:08,670 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.13 2021-05-25 08:50:13,805 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.11 2021-05-25 08:50:18,904 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.1 2021-05-25 08:50:24,026 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.1 2021-05-25 08:50:29,142 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.12 2021-05-25 08:50:34,309 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.13 2021-05-25 08:50:39,405 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.12 2021-05-25 08:50:44,536 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.11 2021-05-25 08:50:49,640 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.13 2021-05-25 08:50:54,746 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.13 2021-05-25 08:50:59,855 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.33 2021-05-25 08:51:04,987 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 71.27 2021-05-25 08:51:10,108 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 73.23 2021-05-25 08:51:15,232 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 74.82 2021-05-25 08:51:20,407 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 75.54 2021-05-25 08:51:25,545 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 75.62 2021-05-25 08:51:30,664 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 76.85 2021-05-25 08:51:35,796 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:51:40,901 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:51:46,009 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 08:51:56,122 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:51:56,244 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 153.620999813 sec 2021-05-25 08:51:56,246 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:51:56,471 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:51:56,674 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.81333333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.65841097352 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 63.2535039915 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 63.0688077989 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.16178602977 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 52.2399253358 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 47.0024500408 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.0558527926 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 16.6624997917 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:51:56,704 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:52:01,823 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 2.49 2021-05-25 08:52:06,992 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 6.28 2021-05-25 08:52:12,115 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 9.94 2021-05-25 08:52:17,262 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 13.06 2021-05-25 08:52:22,411 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 16.15 2021-05-25 08:52:27,549 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 19.3 2021-05-25 08:52:32,690 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 23.13 2021-05-25 08:52:37,842 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 26.24 2021-05-25 08:52:42,980 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 28.97 2021-05-25 08:52:48,131 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-25 08:52:53,272 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:52:58,390 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 08:53:03,519 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:53:08,638 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:53:13,733 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:53:23,943 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:53:24,280 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 87.6089999676 sec 2021-05-25 08:53:24,282 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:53:24,742 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:53:37,325 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.93873026985 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.48333333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 89.1307241747 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 84.4027508847 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.32031067702 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 45.5724262071 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.2747287545 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.56680944682 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:53:37,387 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:53:42,525 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.37 2021-05-25 08:53:47,753 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 4.22 2021-05-25 08:53:52,986 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 8.61 2021-05-25 08:53:58,135 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 12.29 2021-05-25 08:54:03,312 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 16.44 2021-05-25 08:54:10,594 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 20.58 2021-05-25 08:54:15,834 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 22.94 2021-05-25 08:54:23,799 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 31.71 2021-05-25 08:54:28,969 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 08:54:34,111 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 39.89 2021-05-25 08:54:39,250 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:54:44,365 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:54:49,499 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:54:59,661 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:54:59,736 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 82.4170000553 sec 2021-05-25 08:54:59,740 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:54:59,898 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:55:00,121 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 14.4811289356 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.02489917003 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 73.7778962983 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 73.5601963558 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 21.3576214126 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 73.8557951932 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 82.6560885363 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 79.8735335718 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.34864495483 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 8.60680290222 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:55:00,154 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:55:05,309 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 6.33 2021-05-25 08:55:10,477 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 15.62 2021-05-25 08:55:15,678 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 21.79 2021-05-25 08:55:20,855 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 29.14 2021-05-25 08:55:26,007 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 36.17 2021-05-25 08:55:31,184 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:55:36,311 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:55:41,421 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:55:46,506 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:55:51,608 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:56:01,737 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:56:01,868 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 61.7490000725 sec 2021-05-25 08:56:01,868 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 08:56:02,046 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:56:07,723 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 10.784820253 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.55833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 45.3098436615 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.09510158503 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 65.3933333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 45.6233333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 46.0184660511 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.57481417286 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 10.3681605307 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:56:07,769 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:56:12,913 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.57 2021-05-25 08:56:18,071 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 7.08 2021-05-25 08:56:23,240 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 11.64 2021-05-25 08:56:28,401 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 17.49 2021-05-25 08:56:33,556 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 22.92 2021-05-25 08:56:38,710 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 27.43 2021-05-25 08:56:43,911 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.4 2021-05-25 08:56:49,055 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 36.32 2021-05-25 08:56:54,171 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:56:59,295 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 08:57:04,415 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:57:09,507 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:57:14,625 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:57:19,805 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 08:57:30,000 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:57:30,115 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 82.3949999809 sec 2021-05-25 08:57:30,118 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:57:30,309 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:57:30,505 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 11.5114748088 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.50449970002 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 90.5988033932 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 79.9547375778 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 16.9827672411 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 63.9039349344 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 66.2788953517 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 18.8276080471 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 18.9870675288 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:57:30,528 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:57:35,635 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 58.62 2021-05-25 08:57:40,727 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:57:45,831 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:57:50,911 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 08:58:01,019 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:58:01,138 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 30.635999918 sec 2021-05-25 08:58:01,140 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 08:58:01,326 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:58:12,898 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.63674850409 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 19.0369839497 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 40.4183333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 51.9244529495 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 16.3272460188 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 31.1448905779 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 3.46011533718 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 3.28483575821 | Cluster node | | 172.23.107.142 | ['cbas'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:58:12,930 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:58:18,039 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 2.11 2021-05-25 08:58:23,171 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 6.79 2021-05-25 08:58:28,329 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 11.95 2021-05-25 08:58:33,451 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 17.58 2021-05-25 08:58:38,569 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 22.66 2021-05-25 08:58:43,687 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 27.58 2021-05-25 08:58:48,812 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 35.4 2021-05-25 08:58:53,950 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:58:59,078 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 08:59:04,184 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 08:59:09,269 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:59:14,357 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 08:59:19,578 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 08:59:29,729 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 08:59:29,786 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 76.8929998875 sec 2021-05-25 08:59:29,789 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 08:59:29,937 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 08:59:30,144 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 9.63365445515 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 10.3635060584 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 66.8227803797 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 65.6744279071 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.0561638585 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 63.8272695455 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 63.3060551009 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 72.8466982707 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 25.4666666667 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 16.3825142076 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 08:59:30,167 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 08:59:35,316 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 6.25 2021-05-25 08:59:40,493 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 12.74 2021-05-25 08:59:45,653 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 20.39 2021-05-25 08:59:50,798 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 28.83 2021-05-25 08:59:55,930 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:00:01,119 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:00:06,263 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:00:11,385 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:00:16,493 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:00:21,599 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:00:31,733 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:00:31,867 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 61.7260000706 sec 2021-05-25 09:00:31,868 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:00:32,055 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:00:38,091 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.36175602927 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.75160413993 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 65.2583333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 58.7766666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.45348178273 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 60.519341989 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 65.9633333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.2663788937 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 5.75205013668 | Cluster node | | 172.23.105.137 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:00:38,140 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:00:43,298 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:00:48,395 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:00:53,484 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:01:03,595 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:01:03,663 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 25.5770001411 sec 2021-05-25 09:01:03,664 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:01:03,845 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:01:04,068 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.985 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 20.5733333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 37.3272350938 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.6849950757 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.03666666667 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 17.055568519 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 12.8315555778 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 37.189380177 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.01 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 16.4291547911 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:01:04,092 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:01:09,226 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:01:14,322 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:01:19,414 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:01:24,523 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 09:01:34,640 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:01:34,770 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 30.7059998512 sec 2021-05-25 09:01:34,773 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:01:34,963 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:01:41,072 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.78 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.87511458524 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 12.7585459758 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 47.0628369833 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.71635224318 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 16.8469474491 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 12.546532009 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.0427985734 | Cluster node | | 172.23.107.131 | index, n1ql | 7.0.0-5219-enterprise | 6.06363651516 | --- OUT ---> | | 172.23.105.137 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:01:41,114 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:01:46,230 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:01:51,315 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:01:56,404 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:02:01,496 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:02:06,585 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 35.0 2021-05-25 09:02:11,674 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.67 2021-05-25 09:02:16,753 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 09:02:21,838 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:02:26,944 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:02:32,022 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:02:42,128 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:02:42,259 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 61.1919999123 sec 2021-05-25 09:02:42,263 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:02:42,459 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:02:47,650 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 1.46 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.701701751754 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 10.0059996 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 51.6658610977 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.858333333333 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 12.9429019033 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 12.735 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 8.16459394041 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 7.55166666667 | Cluster node | | 172.23.107.131 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+----------------+--------------+ 2021-05-25 09:02:47,707 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:02:52,825 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:02:57,907 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:03:03,006 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:03:08,099 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 62.84 2021-05-25 09:03:13,207 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 68.14 2021-05-25 09:03:18,309 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 69.13 2021-05-25 09:03:23,417 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 71.26 2021-05-25 09:03:28,516 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 71.4 2021-05-25 09:03:33,632 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.75 2021-05-25 09:03:38,723 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.78 2021-05-25 09:03:43,812 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 75.57 2021-05-25 09:03:48,923 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 77.28 2021-05-25 09:03:54,023 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:03:59,105 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:04:04,207 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:04:09,292 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 76.91 2021-05-25 09:04:14,380 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:04:19,474 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:04:24,565 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 76.48 2021-05-25 09:04:29,654 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:04:34,753 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:04:44,859 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:04:44,914 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 117.266000032 sec 2021-05-25 09:04:55,275 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 16 #### 2021-05-25 09:04:55,276 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:04:55,490 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:04:55,730 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 13.3331111148 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.49009150153 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 18.4432796955 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 17.6789226308 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.67327211213 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 14.3009516191 | --- OUT ---> | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 4.99674994583 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 7.68615425638 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.34673911232 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 11.4053801793 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:04:55,766 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:05:00,904 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:05:06,059 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:05:11,289 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:05:16,506 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.66 2021-05-25 09:05:21,650 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.99 2021-05-25 09:05:26,792 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 65.1 2021-05-25 09:05:31,940 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.04 2021-05-25 09:05:37,088 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 67.8 2021-05-25 09:05:42,214 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.15 2021-05-25 09:05:47,326 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.8 2021-05-25 09:05:52,428 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.81 2021-05-25 09:05:57,529 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 71.04 2021-05-25 09:06:02,638 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.19 2021-05-25 09:06:07,792 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.18 2021-05-25 09:06:12,894 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.19 2021-05-25 09:06:18,017 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 75.77 2021-05-25 09:06:23,148 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:06:28,249 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:06:38,361 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:06:38,500 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 102.776000023 sec 2021-05-25 09:06:38,503 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:06:38,674 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:06:38,881 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.71666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.03340055668 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 55.2324127931 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.0591323478 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.21654639089 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 21.585359756 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 51.8233333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 18.4875910462 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 43.0216666667 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:06:38,917 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:06:44,036 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:06:49,157 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 3.85 2021-05-25 09:06:54,358 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 8.58 2021-05-25 09:06:59,517 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 12.23 2021-05-25 09:07:04,730 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 16.25 2021-05-25 09:07:09,950 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 20.52 2021-05-25 09:07:15,134 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 23.58 2021-05-25 09:07:20,302 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 26.43 2021-05-25 09:07:25,437 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 29.73 2021-05-25 09:07:30,551 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 32.56 2021-05-25 09:07:35,703 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:07:40,832 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:07:45,940 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:07:51,030 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:07:56,164 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:08:06,296 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:08:06,437 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 87.5590000153 sec 2021-05-25 09:08:06,440 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:08:06,608 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:08:19,219 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 10.8113062898 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.59028634288 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 95.9570667869 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.955 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 14.2933333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 93.9269162805 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.7559788677 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 58.6429321466 | Cluster node | | 172.23.107.127 | ['kv'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:08:19,280 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:08:24,424 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.89 2021-05-25 09:08:29,696 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 3.01 2021-05-25 09:08:34,915 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 6.28 2021-05-25 09:08:40,153 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 7.95 2021-05-25 09:08:45,325 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 9.24 2021-05-25 09:08:50,628 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 14.68 2021-05-25 09:08:55,933 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.15 2021-05-25 09:09:01,115 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 16.02 2021-05-25 09:09:06,269 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 21.1 2021-05-25 09:09:11,500 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 26.73 2021-05-25 09:09:16,763 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 32.12 2021-05-25 09:09:22,023 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 09:09:27,165 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 39.27 2021-05-25 09:09:32,303 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:09:37,398 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:09:42,517 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:09:52,671 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:09:52,732 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 93.5160000324 sec 2021-05-25 09:09:52,733 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:09:52,911 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:09:53,134 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.28472384254 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.460135530829 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 72.8883333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.1464426779 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.5282078632 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 71.3288111865 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 9.55801473284 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 71.0414479276 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.8205910296 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 62.11 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 09:09:53,158 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:09:58,331 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 6.57 2021-05-25 09:10:03,556 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 14.45 2021-05-25 09:10:08,737 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 24.06 2021-05-25 09:10:13,959 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 33.35 2021-05-25 09:10:19,128 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 39.84 2021-05-25 09:10:24,319 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:10:29,447 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:10:34,562 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:10:39,684 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:10:44,812 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:10:54,950 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:10:55,091 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 61.9609999657 sec 2021-05-25 09:10:55,092 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:10:55,282 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:11:01,308 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 14.4014266429 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.19656339061 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 58.543048565 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.6485108085 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 48.3058050968 | --- OUT ---> | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 13.9256143162 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 62.5486170231 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 20.3636727279 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 46.51 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:11:01,335 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:11:06,460 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.57 2021-05-25 09:11:11,668 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 8.71 2021-05-25 09:11:16,861 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 13.1 2021-05-25 09:11:22,029 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 17.72 2021-05-25 09:11:27,164 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 22.57 2021-05-25 09:11:32,351 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 27.43 2021-05-25 09:11:37,516 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.93 2021-05-25 09:11:42,671 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.96 2021-05-25 09:11:47,816 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.06 2021-05-25 09:11:52,960 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:11:58,086 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:12:03,203 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:12:08,295 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:12:13,418 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:12:23,542 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:12:23,667 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 82.3630001545 sec 2021-05-25 09:12:23,670 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:12:23,834 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:12:24,028 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 8.64666666667 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.41666666667 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 73.6212270205 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 91.6147204907 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 12.894570181 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 15.574740421 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 71.9858451986 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 7.08488191863 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 73.0733333333 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:12:24,051 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:12:29,174 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.44 2021-05-25 09:12:34,266 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:12:39,365 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:12:44,457 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 77.78 2021-05-25 09:12:54,572 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:12:54,709 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 30.6840000153 sec 2021-05-25 09:12:54,710 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:12:54,885 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:13:06,362 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.70657155714 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 37.942700955 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 12.3137437915 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 19.7156808826 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 17.4480425326 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 32.5284965471 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.1311622946 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 27.2921215354 | Cluster node | | 172.23.107.142 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:13:06,394 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:13:11,536 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 3.28 2021-05-25 09:13:16,697 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 8.59 2021-05-25 09:13:21,858 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 16.0 2021-05-25 09:13:27,009 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 22.25 2021-05-25 09:13:32,154 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 27.49 2021-05-25 09:13:37,288 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 34.82 2021-05-25 09:13:42,454 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:13:47,565 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:13:52,648 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 64.54 2021-05-25 09:13:57,730 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:14:02,832 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:14:07,961 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:14:18,088 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:14:18,164 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 71.8059999943 sec 2021-05-25 09:14:18,167 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:14:18,377 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:14:18,657 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.11506858448 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.40333333333 | Cluster node | | 172.23.107.127 | kv | 7.0.0-5219-enterprise | 37.1872864548 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 38.8297251804 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.06333333333 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 60.2103403447 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 10.9144542773 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 44.3555097015 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.8853628454 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 54.5833333333 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:14:18,701 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:14:23,859 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 6.72 2021-05-25 09:14:29,108 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 15.94 2021-05-25 09:14:34,289 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 24.46 2021-05-25 09:14:39,487 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 31.1 2021-05-25 09:14:44,698 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 38.44 2021-05-25 09:14:49,885 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:14:54,996 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:15:00,118 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:15:05,230 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:15:10,364 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:15:20,523 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:15:20,642 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 61.9889998436 sec 2021-05-25 09:15:20,644 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:15:20,818 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:15:26,328 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 0.911666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.570019000633 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 56.2502083403 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.04329855671 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 54.8440859319 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 2.5 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 56.329789007 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.55649623358 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 65.6266666667 | Cluster node | | 172.23.107.127 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+----------------+--------------+ 2021-05-25 09:15:26,355 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:15:31,477 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:15:36,575 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:15:41,680 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:15:51,798 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:15:51,871 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 25.5480000973 sec 2021-05-25 09:15:51,872 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:15:52,043 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:15:52,266 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.58677644627 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.493349778326 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.56819053016 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 19.1916666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.71016183603 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 36.8995633188 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 9.81849697495 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 63.9349176478 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.81161980634 | --- OUT ---> | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 20.625 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 09:15:52,306 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:15:57,437 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:16:02,528 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:16:07,611 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:16:12,700 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:16:17,788 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:16:22,872 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.55 2021-05-25 09:16:27,982 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 31.55 2021-05-25 09:16:33,111 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.1 2021-05-25 09:16:38,217 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.62 2021-05-25 09:16:43,309 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 34.66 2021-05-25 09:16:48,405 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.21 2021-05-25 09:16:53,493 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 36.21 2021-05-25 09:16:58,609 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 37.76 2021-05-25 09:17:03,727 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 39.31 2021-05-25 09:17:08,848 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.86 2021-05-25 09:17:13,937 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.86 2021-05-25 09:17:19,048 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 42.41 2021-05-25 09:17:24,153 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.97 2021-05-25 09:17:29,255 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 09:17:34,334 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:17:39,420 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:17:44,513 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:17:54,628 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:17:54,776 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 122.512000084 sec 2021-05-25 09:17:54,779 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:17:54,969 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:18:00,960 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.184880252 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.25842097368 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 12.2147964201 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 17.0311143754 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.9814339522 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 38.6723112052 | Cluster node | | 172.23.105.137 | index, n1ql | 7.0.0-5219-enterprise | 11.6370545685 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 38.2570580981 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 16.1983333333 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:18:00,993 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:18:06,095 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:18:11,178 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:18:16,276 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:18:21,375 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 09:18:26,464 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 09:18:31,578 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 33.13 2021-05-25 09:18:36,686 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 09:18:41,776 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 09:18:46,878 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 09:18:51,964 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:18:57,052 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:19:02,167 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 09:19:07,260 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 09:19:12,345 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 09:19:17,447 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-25 09:19:22,546 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-25 09:19:27,654 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 09:19:32,755 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:19:37,848 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:19:42,944 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:19:53,053 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:19:53,187 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 112.230999947 sec 2021-05-25 09:19:53,190 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:19:53,362 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:19:59,424 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 2.00170002833 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.28327861202 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.6783333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 66.7450039169 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.61658583737 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 57.2907118215 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 63.4627243787 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 12.2693865307 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 24.9766666667 | Cluster node | | 172.23.105.137 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 09:19:59,482 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:20:04,632 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:20:09,723 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:20:14,846 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:20:19,987 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 62.79 2021-05-25 09:20:25,142 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 68.24 2021-05-25 09:20:30,283 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 71.54 2021-05-25 09:20:35,434 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 75.25 2021-05-25 09:20:40,543 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 09:20:45,632 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.96 2021-05-25 09:20:50,719 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:20:55,809 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 77.07 2021-05-25 09:21:00,905 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:21:06,010 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:21:11,104 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:21:16,197 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:21:21,293 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:21:26,381 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:21:31,505 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 75.32 2021-05-25 09:21:36,621 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:21:41,710 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:21:51,822 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:21:51,882 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 112.460999966 sec 2021-05-25 09:22:02,691 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 17 #### 2021-05-25 09:22:02,694 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:22:02,969 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:22:03,308 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 3.27827869536 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.00511675195 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 6.11833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 21.0946484225 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.30511017034 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 41.4602846572 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 11.4570485683 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 23.0444855576 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.6917781963 | Cluster node | | 172.23.107.131 | eventing | 7.0.0-5219-enterprise | 15.1035850598 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:22:03,334 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:22:08,483 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:22:13,598 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:22:18,822 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:22:24,006 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 61.96 2021-05-25 09:22:29,236 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 65.55 2021-05-25 09:22:34,387 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.8 2021-05-25 09:22:39,601 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.78 2021-05-25 09:22:44,775 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.79 2021-05-25 09:22:49,914 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.82 2021-05-25 09:22:55,095 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.8 2021-05-25 09:23:00,233 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.79 2021-05-25 09:23:05,400 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 67.85 2021-05-25 09:23:10,528 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 68.22 2021-05-25 09:23:15,696 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 71.18 2021-05-25 09:23:20,805 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 74.52 2021-05-25 09:23:25,983 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:23:31,095 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:23:36,198 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:23:46,312 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:23:46,430 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 103.128000021 sec 2021-05-25 09:23:46,433 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:23:46,612 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:23:46,848 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 6.5115581407 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.67833333333 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.6278124063 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 67.3677544082 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.40822652956 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 67.0822360745 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 39.765 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 76.2389624263 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.5291597913 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:23:46,914 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:23:52,052 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 3.02 2021-05-25 09:23:57,206 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 8.24 2021-05-25 09:24:02,391 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 12.64 2021-05-25 09:24:07,615 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 16.54 2021-05-25 09:24:12,822 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 20.76 2021-05-25 09:24:18,013 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 25.1 2021-05-25 09:24:23,210 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 28.25 2021-05-25 09:24:28,463 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 31.77 2021-05-25 09:24:33,732 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:24:38,880 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:24:44,020 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:24:49,154 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:24:54,295 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:25:04,410 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:25:04,545 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 77.6980001926 sec 2021-05-25 09:25:04,546 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:25:04,740 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:25:15,118 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 4.22 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 6.015 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 11.7179427352 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.42825952901 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 78.1247916806 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 40.6259895668 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 76.4481020265 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.7001783363 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | | 172.23.107.131 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:25:15,161 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:25:20,331 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.54 2021-05-25 09:25:25,559 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 4.94 2021-05-25 09:25:30,861 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 8.31 2021-05-25 09:25:36,157 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 11.51 2021-05-25 09:25:41,339 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 17.49 2021-05-25 09:25:46,568 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 20.97 2021-05-25 09:25:51,756 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 26.64 2021-05-25 09:25:56,880 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 32.97 2021-05-25 09:26:02,059 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 36.93 2021-05-25 09:26:07,191 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:26:12,301 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:26:17,414 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:26:22,530 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 80.0 2021-05-25 09:26:32,680 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:26:32,740 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 77.628000021 sec 2021-05-25 09:26:32,743 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:26:32,990 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:26:33,243 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 7.52512541876 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.570019000633 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 14.1414309762 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 75.8612643544 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.48188272942 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 65.7805260175 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 71.3821436309 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 60.2606623223 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.535 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 61.7473042116 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 09:26:33,278 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:26:38,464 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-05-25 09:26:43,625 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 12.34 2021-05-25 09:26:48,799 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 19.61 2021-05-25 09:26:54,039 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 27.97 2021-05-25 09:26:59,214 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.01 2021-05-25 09:27:04,342 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:27:09,484 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:27:14,615 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:27:19,737 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:27:24,835 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:27:34,957 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:27:35,505 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 62.2650001049 sec 2021-05-25 09:27:35,506 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:27:35,694 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:27:41,411 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 10.9073938263 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.34654422426 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 19.3976365485 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.6236874562 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 84.1949753777 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 62.7877131286 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 84.1720724938 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 18.408640144 | Cluster node | | 172.23.107.131 | kv | 7.0.0-5219-enterprise | 91.7334401709 | --- OUT ---> | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:27:41,467 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:27:46,602 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 1.94 2021-05-25 09:27:51,767 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 7.45 2021-05-25 09:27:56,944 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 13.49 2021-05-25 09:28:02,160 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 19.12 2021-05-25 09:28:07,361 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 23.23 2021-05-25 09:28:12,542 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 28.5 2021-05-25 09:28:17,716 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.19 2021-05-25 09:28:22,895 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 38.77 2021-05-25 09:28:28,032 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:28:33,144 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:28:38,266 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:28:43,401 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:28:48,506 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:28:58,660 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:28:58,770 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 77.3650000095 sec 2021-05-25 09:28:58,773 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:28:58,936 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:28:59,144 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | cbas | 7.0.0-5219-enterprise | 5.5920394693 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.37003950066 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 4.40166666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 83.1233333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.45492575124 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 66.2233333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 64.0104670156 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 68.6460451318 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.79309367865 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:28:59,168 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:29:04,279 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 56.83 2021-05-25 09:29:09,392 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:29:14,487 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:29:19,588 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 83.33 2021-05-25 09:29:29,742 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:29:29,875 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 30.7339999676 sec 2021-05-25 09:29:29,878 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:29:30,081 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:29:40,812 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.18894592973 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 15.1821727391 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 15.805 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.52314922836 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 31.3278112031 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 15.730262171 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 30.519350645 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.1985199753 | Cluster node | | 172.23.107.131 | ['cbas'] | | | <--- IN --- | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:29:40,862 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:29:46,048 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 1.87 2021-05-25 09:29:51,178 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 7.72 2021-05-25 09:29:56,339 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 14.13 2021-05-25 09:30:01,483 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 19.28 2021-05-25 09:30:06,599 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 23.42 2021-05-25 09:30:11,717 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 29.04 2021-05-25 09:30:16,816 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 32.16 2021-05-25 09:30:21,946 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 37.88 2021-05-25 09:30:27,072 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:30:32,190 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:30:37,276 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 09:30:42,375 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:30:47,480 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:30:57,611 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:30:57,665 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 76.8579998016 sec 2021-05-25 09:30:57,667 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:30:57,858 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:30:58,066 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 69.505 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 17.0313828103 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 5.02675044584 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 52.7333333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.79677994633 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 40.6533333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 35.1550910879 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 46.7483333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.72685756192 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 17.524503686 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:30:58,095 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:31:03,253 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 6.56 2021-05-25 09:31:08,499 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 13.98 2021-05-25 09:31:13,674 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 21.87 2021-05-25 09:31:18,881 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 30.24 2021-05-25 09:31:24,117 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 37.73 2021-05-25 09:31:29,309 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:31:34,424 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:31:39,569 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:31:44,733 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:31:49,887 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:32:00,005 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:32:00,138 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 62.0740001202 sec 2021-05-25 09:32:00,138 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:32:00,336 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:32:06,028 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 71.5904734921 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.90529684323 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 74.348760854 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.73474217526 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 81.9283333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 75.0791693056 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 72.2633333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.38301418262 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 1.80109501235 | Cluster node | | 172.23.107.142 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:32:06,065 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:32:11,200 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:32:16,292 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:32:21,380 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:32:31,507 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:32:31,565 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 25.5420000553 sec 2021-05-25 09:32:31,568 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:32:31,759 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:32:31,960 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 22.1879635339 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 77.5262921049 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 1.405 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 30.025 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.17168619477 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 12.2133333333 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 11.1201853364 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 31.2493754788 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.66338877963 | --- OUT ---> | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 8.98348305805 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:32:31,986 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:32:37,094 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:32:42,186 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:32:47,305 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:32:52,414 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.21 2021-05-25 09:32:57,503 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.21 2021-05-25 09:33:02,651 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 36.43 2021-05-25 09:33:07,742 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:33:12,832 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 39.64 2021-05-25 09:33:17,933 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 09:33:23,043 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:33:28,141 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:33:38,302 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:33:38,427 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 66.4700000286 sec 2021-05-25 09:33:38,428 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:33:38,630 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:33:44,776 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.5353845128 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 77.7587040216 | Cluster node | | 172.23.107.127 | index, n1ql | 7.0.0-5219-enterprise | 8.23292168725 | --- OUT ---> | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 49.0883637212 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.45659239013 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 53.2808880148 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 36.3272721212 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 56.5342755713 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.42507375123 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:33:44,845 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:33:49,987 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:33:55,085 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:34:00,165 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:34:05,252 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 09:34:10,348 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 09:34:15,437 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 09:34:20,542 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 09:34:25,644 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 09:34:30,746 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:34:35,858 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:34:40,956 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 09:34:46,046 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 09:34:51,157 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 09:34:56,247 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 42.5 2021-05-25 09:35:01,362 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-25 09:35:06,496 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:35:11,585 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:35:16,687 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:35:26,792 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:35:26,914 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 102.142999887 sec 2021-05-25 09:35:26,917 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:35:27,107 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:35:32,573 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.7168619477 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.0215659478 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.5372756213 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.84666666667 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 16.4819413657 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 24.5320755346 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 43.3066666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.8735312255 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.23640484642 | Cluster node | | 172.23.107.127 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 09:35:32,628 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:35:37,730 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:35:42,842 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:35:47,956 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:35:53,109 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 63.02 2021-05-25 09:35:58,243 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 64.72 2021-05-25 09:36:03,385 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 68.62 2021-05-25 09:36:08,572 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 69.62 2021-05-25 09:36:13,709 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.83 2021-05-25 09:36:18,796 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.87 2021-05-25 09:36:23,927 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 75.65 2021-05-25 09:36:29,042 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 74.75 2021-05-25 09:36:34,145 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:36:39,246 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 09:36:44,348 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:36:49,461 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 78.58 2021-05-25 09:36:54,578 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:36:59,674 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:37:04,767 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:37:09,869 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:37:14,976 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:37:25,108 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:37:25,177 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 112.606999874 sec 2021-05-25 09:37:35,539 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 18 #### 2021-05-25 09:37:35,542 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:37:35,762 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:37:36,009 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 9.16666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.31833333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 17.1080481992 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 35.4155902598 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.205 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 29.3466666667 | Cluster node | | 172.23.105.137 | eventing | 7.0.0-5219-enterprise | 21.1983333333 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 47.4500849972 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.36662722288 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 2.39507983599 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:37:36,036 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:37:41,165 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:37:46,286 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:37:51,453 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:37:56,634 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 60.95 2021-05-25 09:38:01,782 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 63.15 2021-05-25 09:38:06,921 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 64.39 2021-05-25 09:38:12,075 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 64.87 2021-05-25 09:38:17,290 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 66.5 2021-05-25 09:38:22,500 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 67.97 2021-05-25 09:38:27,661 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.7 2021-05-25 09:38:32,871 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 68.72 2021-05-25 09:38:38,056 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 69.58 2021-05-25 09:38:43,187 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 72.62 2021-05-25 09:38:48,312 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 77.0 2021-05-25 09:38:53,459 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.17 2021-05-25 09:38:58,575 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:39:03,686 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:39:08,799 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:39:13,937 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:39:24,091 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:39:24,226 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 108.223999977 sec 2021-05-25 09:39:24,230 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:39:24,427 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:39:24,665 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 5.17867857857 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 25.3133333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 62.6748572764 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 69.0634531727 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 14.5028752396 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 80.5726857562 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 78.7253545774 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.8427385754 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.70098325626 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:39:24,691 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:39:29,848 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 3.26 2021-05-25 09:39:35,012 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 7.55 2021-05-25 09:39:40,128 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 11.08 2021-05-25 09:39:45,329 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 14.92 2021-05-25 09:39:50,510 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 18.35 2021-05-25 09:39:55,690 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 21.53 2021-05-25 09:40:00,841 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 23.87 2021-05-25 09:40:06,006 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 27.08 2021-05-25 09:40:11,190 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:40:16,371 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 32.74 2021-05-25 09:40:21,556 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:40:26,690 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:40:31,802 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:40:36,892 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:40:42,000 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:40:52,164 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:40:52,315 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 87.6529998779 sec 2021-05-25 09:40:52,319 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:40:52,503 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:41:03,839 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 22.1616666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.988366278876 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 59.4208700294 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 77.1454245919 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.3901731696 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 73.2976103308 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 3.28322389254 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 9.34817753037 | Cluster node | | 172.23.104.247 | ['kv'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 09:41:03,864 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:41:09,052 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 1.01 2021-05-25 09:41:14,279 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 6.09 2021-05-25 09:41:19,470 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 7.34 2021-05-25 09:41:24,739 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 14.35 2021-05-25 09:41:29,943 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 14.8 2021-05-25 09:41:35,167 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 20.8 2021-05-25 09:41:40,346 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 26.08 2021-05-25 09:41:45,607 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 33.13 2021-05-25 09:41:50,802 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 37.38 2021-05-25 09:41:56,085 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:42:01,211 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:42:06,319 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:42:16,483 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:42:16,555 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 72.7189998627 sec 2021-05-25 09:42:16,556 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:42:16,776 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:42:17,101 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 21.7179713671 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.11484808587 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 81.849697495 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 76.9516666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 13.9078697377 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 78.83 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 76.1983333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 74.5706583239 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.5480741988 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 14.7989151234 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:42:17,148 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:42:22,275 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 5.39 2021-05-25 09:42:27,483 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 12.34 2021-05-25 09:42:32,723 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 18.83 2021-05-25 09:42:37,923 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 25.54 2021-05-25 09:42:43,118 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.95 2021-05-25 09:42:48,292 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 09:42:53,430 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:42:58,571 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:43:03,687 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:43:08,805 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:43:14,000 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:43:24,130 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:43:24,280 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 67.1829998493 sec 2021-05-25 09:43:24,282 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:43:24,480 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 09:43:30,322 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 22.4216666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 12.7866666667 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 56.5107248213 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 77.6296271605 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.83 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 64.4288142938 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 73.8542048598 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.0485841431 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 16.5873464905 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:43:30,367 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:43:35,487 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:43:41,312 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.29 2021-05-25 09:43:46,469 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 6.61 2021-05-25 09:43:51,724 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 12.57 2021-05-25 09:43:56,884 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 18.13 2021-05-25 09:44:02,088 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 23.22 2021-05-25 09:44:07,315 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 28.83 2021-05-25 09:44:12,473 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 34.39 2021-05-25 09:44:17,637 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 38.89 2021-05-25 09:44:22,803 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:44:27,940 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:44:33,055 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:44:38,154 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:44:43,283 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:44:53,447 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:44:53,592 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 83.2739999294 sec 2021-05-25 09:44:53,595 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:44:53,778 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:44:54,005 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 25.6816212253 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.02 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 60.9373177114 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 61.8476974616 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.55381102388 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 55.0768358945 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 71.5883333333 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 25.1787529792 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.45178359791 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:44:54,033 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:44:59,168 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:45:04,259 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-25 09:45:09,348 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:45:14,440 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:45:19,520 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:45:29,663 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:45:29,778 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 35.7760000229 sec 2021-05-25 09:45:29,779 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:45:29,973 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:45:41,466 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 7.34345572426 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.90339838997 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 24.9216666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 68.1483333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 62.5256245729 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 47.03 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.52148261725 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 47.038855146 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.104.247 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:45:41,490 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:45:46,661 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.5 2021-05-25 09:45:51,842 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.72 2021-05-25 09:45:56,989 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 14.4 2021-05-25 09:46:05,276 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 20.09 2021-05-25 09:46:10,434 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 23.51 2021-05-25 09:46:15,596 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 27.84 2021-05-25 09:46:20,756 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.86 2021-05-25 09:46:25,903 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 36.31 2021-05-25 09:46:31,036 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:46:36,148 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:46:41,244 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 09:46:46,335 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:46:51,447 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:46:56,984 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 09:47:07,193 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:47:07,272 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 85.8099999428 sec 2021-05-25 09:47:07,273 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:47:07,484 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:47:07,743 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 9.94833333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.43833333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 64.2928685897 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 56.0769192248 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.19326344561 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 67.3211220187 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 53.3634906857 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 61.421607906 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.555 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.6466058899 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:47:07,779 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:47:12,974 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 3.75 2021-05-25 09:47:18,164 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 10.47 2021-05-25 09:47:23,371 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 18.28 2021-05-25 09:47:28,552 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 26.02 2021-05-25 09:47:33,737 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 34.45 2021-05-25 09:47:38,940 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:47:44,073 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 09:47:49,187 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 09:47:54,279 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:47:59,384 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 09:48:04,539 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 09:48:14,724 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:48:14,851 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 67.111000061 sec 2021-05-25 09:48:14,851 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:48:15,030 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:48:20,930 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 17.085812376 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 9.00833333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 53.1498950035 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 73.0958968034 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.38180636344 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 36.69 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 42.5 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.5846735888 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 8.14234922966 | Cluster node | | 172.23.104.247 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:48:20,990 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:48:26,118 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:48:31,221 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:48:36,325 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:48:46,466 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:48:46,548 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 25.621999979 sec 2021-05-25 09:48:46,549 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:48:46,770 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:48:47,000 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 7.96679944666 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.76162063966 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 19.1403190053 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 28.255 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.31666666667 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 7.56518545722 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 20.7803463391 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 20.4920082001 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 7.76153730771 | --- OUT ---> | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.4665590521 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:48:47,049 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:48:52,342 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:48:57,437 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:49:02,532 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:49:07,631 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 33.46 2021-05-25 09:49:12,726 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 09:49:17,831 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 36.92 2021-05-25 09:49:22,926 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.38 2021-05-25 09:49:28,022 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.38 2021-05-25 09:49:33,108 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 43.85 2021-05-25 09:49:38,191 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:49:43,293 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:49:48,401 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:49:58,515 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:49:58,644 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 71.6480000019 sec 2021-05-25 09:49:58,647 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 09:49:58,855 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:50:04,605 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 5.44157597373 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.54166666667 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 23.5153919232 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 10.1285021417 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.74685400937 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 4.74 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 31.7705295088 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 10.1566666667 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.87686050969 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 09:50:04,716 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:50:09,871 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 09:50:14,953 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:50:20,086 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 09:50:25,188 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.8 2021-05-25 09:50:30,273 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.8 2021-05-25 09:50:35,358 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 31.8 2021-05-25 09:50:40,450 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 33.6 2021-05-25 09:50:45,575 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.4 2021-05-25 09:50:50,667 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 36.0 2021-05-25 09:50:55,753 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.2 2021-05-25 09:51:00,849 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 37.2 2021-05-25 09:51:05,969 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 09:51:11,048 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.2 2021-05-25 09:51:16,148 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.8 2021-05-25 09:51:21,239 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 42.6 2021-05-25 09:51:26,338 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 42.6 2021-05-25 09:51:31,430 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 43.8 2021-05-25 09:51:36,516 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 09:51:41,638 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:51:46,729 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 09:51:56,838 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:51:56,963 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 112.363000154 sec 2021-05-25 09:51:56,964 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:51:57,177 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:52:03,066 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.90660155664 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 24.2729287845 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 19.2716666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.10965935604 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 14.17415721 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 30.8623045898 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 12.6018766979 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.7283333333 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.84325261246 | Cluster node | | 172.23.107.142 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 09:52:03,153 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:52:08,303 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:52:13,401 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:52:18,539 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:52:23,647 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 62.79 2021-05-25 09:52:28,769 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 67.51 2021-05-25 09:52:33,895 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 67.37 2021-05-25 09:52:39,013 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 69.01 2021-05-25 09:52:44,125 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 71.96 2021-05-25 09:52:49,230 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.92 2021-05-25 09:52:54,349 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.94 2021-05-25 09:52:59,444 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:04,532 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 75.79 2021-05-25 09:53:09,621 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:14,727 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:19,819 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:24,923 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 78.89 2021-05-25 09:53:30,015 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:35,117 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 09:53:40,223 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:53:45,323 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 76.96 2021-05-25 09:53:50,423 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:54:00,568 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:54:00,625 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 117.559999943 sec 2021-05-25 09:54:11,220 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 19 #### 2021-05-25 09:54:11,221 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:54:11,423 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:54:11,698 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | eventing | 7.0.0-5219-enterprise | 20.83 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.05678427974 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 13.6478784041 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 44.0364678823 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.085 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 9.42666666667 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 58.3343055718 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 63.4092272818 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.84650255829 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.08034213636 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:54:11,739 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:54:17,000 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:54:22,105 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:54:27,220 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 09:54:35,342 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 62.29 2021-05-25 09:54:40,487 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 64.11 2021-05-25 09:54:45,618 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 65.71 2021-05-25 09:54:50,736 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 66.3 2021-05-25 09:54:55,868 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 67.67 2021-05-25 09:55:00,999 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.04 2021-05-25 09:55:06,125 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.01 2021-05-25 09:55:11,256 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.03 2021-05-25 09:55:16,377 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.03 2021-05-25 09:55:21,497 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.04 2021-05-25 09:55:26,631 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.05 2021-05-25 09:55:31,766 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.02 2021-05-25 09:55:36,898 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.03 2021-05-25 09:55:42,013 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.03 2021-05-25 09:55:47,118 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.0 2021-05-25 09:55:52,230 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.05 2021-05-25 09:55:57,348 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.04 2021-05-25 09:56:02,450 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.01 2021-05-25 09:56:07,579 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.04 2021-05-25 09:56:12,703 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.05 2021-05-25 09:56:17,818 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.13 2021-05-25 09:56:22,933 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 69.69 2021-05-25 09:56:28,042 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.99 2021-05-25 09:56:33,177 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 72.71 2021-05-25 09:56:38,306 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 72.9 2021-05-25 09:56:43,417 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 73.57 2021-05-25 09:56:48,538 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 73.63 2021-05-25 09:56:53,660 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 73.64 2021-05-25 09:56:58,769 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 74.42 2021-05-25 09:57:03,898 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 75.23 2021-05-25 09:57:09,023 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 76.01 2021-05-25 09:57:14,138 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 78.27 2021-05-25 09:57:19,239 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:57:24,342 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:57:29,466 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 09:57:39,598 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:57:39,717 | test | INFO | pool-2-thread-7 | [task:check:414] Rebalance completed with progress: 100% in 208.023999929 sec 2021-05-25 09:57:39,720 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 09:57:39,903 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:57:40,130 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.42024734158 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 53.335 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 63.5662144595 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 9.03939737351 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 12.0910573435 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 71.9523984133 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 79.5553407443 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.6001331226 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 8.28305723143 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:57:40,168 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:57:45,434 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 3.04 2021-05-25 09:57:50,595 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 7.63 2021-05-25 09:57:55,782 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 11.33 2021-05-25 09:58:00,930 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 15.09 2021-05-25 09:58:06,061 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 18.27 2021-05-25 09:58:11,233 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 21.83 2021-05-25 09:58:16,365 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 25.2 2021-05-25 09:58:21,505 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 28.5 2021-05-25 09:58:26,687 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 31.01 2021-05-25 09:58:31,818 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:58:36,940 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:58:42,088 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 09:58:47,240 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:58:52,345 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 09:58:57,706 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 83.33 2021-05-25 09:59:08,065 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 09:59:08,188 | test | INFO | pool-2-thread-8 | [task:check:414] Rebalance completed with progress: 100% in 88.0619997978 sec 2021-05-25 09:59:08,191 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 09:59:08,423 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 09:59:20,257 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.20526026301 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 53.1498950035 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.28675477925 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 14.9761674611 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 92.3448724145 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 91.4436895959 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.0663988934 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.51990800153 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | | 172.23.107.142 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 09:59:20,372 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 09:59:25,528 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.26 2021-05-25 09:59:30,684 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 4.27 2021-05-25 09:59:35,851 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 9.09 2021-05-25 09:59:41,043 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 12.61 2021-05-25 09:59:46,213 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 16.33 2021-05-25 09:59:51,394 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 18.52 2021-05-25 09:59:56,581 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 23.18 2021-05-25 10:00:01,757 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 28.55 2021-05-25 10:00:06,913 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 33.31 2021-05-25 10:00:12,124 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 36.6 2021-05-25 10:00:17,269 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 39.89 2021-05-25 10:00:22,430 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:00:27,530 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:00:32,645 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:00:42,770 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:00:42,834 | test | INFO | pool-2-thread-9 | [task:check:414] Rebalance completed with progress: 100% in 82.5810000896 sec 2021-05-25 10:00:42,836 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:00:43,019 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:00:43,255 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 69.7378289638 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.46333333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 77.0312838547 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 70.0016666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.99653339111 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 30.1073297557 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 44.9707495125 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 60.05 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.37166666667 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 7.24 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:00:43,280 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:00:48,513 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 2.58 2021-05-25 10:00:53,683 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 10.45 2021-05-25 10:00:58,884 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 15.69 2021-05-25 10:01:04,016 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.92 2021-05-25 10:01:09,190 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 29.68 2021-05-25 10:01:14,344 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 35.54 2021-05-25 10:01:19,493 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:01:24,611 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:01:29,749 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:01:34,888 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:01:39,982 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:01:45,091 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:01:55,219 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:01:55,361 | test | INFO | pool-2-thread-10 | [task:check:414] Rebalance completed with progress: 100% in 72.1100001335 sec 2021-05-25 10:01:55,361 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:01:55,549 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:02:01,243 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 50.65413396 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 22.7824260809 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 56.3283333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 59.1930268991 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.99026634221 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 30.9923002567 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 35.5733333333 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 19.6106861324 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 8.63061950153 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:02:01,318 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:02:06,492 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:02:11,671 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 2.93 2021-05-25 10:02:16,818 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 8.74 2021-05-25 10:02:22,055 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 14.13 2021-05-25 10:02:27,210 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 18.94 2021-05-25 10:02:32,358 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 25.1 2021-05-25 10:02:37,519 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.73 2021-05-25 10:02:42,683 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.43 2021-05-25 10:02:47,864 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:02:52,983 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:02:58,118 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:03:03,229 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:03:08,323 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:03:13,415 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:03:23,540 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:03:23,684 | test | INFO | pool-2-thread-11 | [task:check:414] Rebalance completed with progress: 100% in 82.4459998608 sec 2021-05-25 10:03:23,687 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:03:23,911 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:03:24,115 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 60.1039982667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.24028268551 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 76.8948330287 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 61.198059903 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 3.67321089297 | --- OUT ---> | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 25.3662438959 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 78.3655606217 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 14.505 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.91106214633 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:03:24,145 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:03:29,388 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.44 2021-05-25 10:03:34,473 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:03:39,562 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:03:44,677 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:03:54,786 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:03:54,923 | test | INFO | pool-2-thread-12 | [task:check:414] Rebalance completed with progress: 100% in 30.8100001812 sec 2021-05-25 10:03:54,924 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:03:55,098 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:04:07,497 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 31.3783333333 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.48839147319 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 28.9804830081 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 34.6089101485 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 12.0033333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 37.8389639827 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.0013332889 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 14.79 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:04:07,523 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:04:12,651 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 3.5 2021-05-25 10:04:17,760 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 11.76 2021-05-25 10:04:22,869 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 16.4 2021-05-25 10:04:27,969 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 19.91 2021-05-25 10:04:33,078 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 24.48 2021-05-25 10:04:38,190 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 29.06 2021-05-25 10:04:43,279 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 33.01 2021-05-25 10:04:48,378 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 36.65 2021-05-25 10:04:53,506 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 10:04:58,631 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 10:05:03,723 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 69.8 2021-05-25 10:05:08,809 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:05:13,908 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:05:24,052 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:05:24,121 | test | INFO | pool-2-thread-13 | [task:check:414] Rebalance completed with progress: 100% in 76.6259999275 sec 2021-05-25 10:05:24,122 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:05:24,342 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:05:24,607 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 71.3278554643 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.42491916936 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 60.9461843112 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 70.1283333333 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 11.4066666667 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 13.2268871148 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 35.5195173172 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 81.9801021601 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 16.6680555324 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.34662059526 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:05:24,638 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:05:29,908 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 4.3 2021-05-25 10:05:35,072 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 12.42 2021-05-25 10:05:40,296 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 19.06 2021-05-25 10:05:45,443 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 26.64 2021-05-25 10:05:50,602 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 34.53 2021-05-25 10:05:55,786 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:06:00,921 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:06:06,039 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 10:06:11,132 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:06:16,226 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:06:26,444 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:06:27,147 | test | INFO | pool-2-thread-14 | [task:check:414] Rebalance completed with progress: 100% in 62.5419998169 sec 2021-05-25 10:06:27,148 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:06:27,346 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:06:34,104 | test | INFO | pool-2-thread-15 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 62.4299003212 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.40333333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 50.4958415974 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.77576251726 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 8.32075377801 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 68.1777274091 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 65.6516666667 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.50054830027 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.79732090856 | Cluster node | | 172.23.107.129 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 10:06:34,135 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:06:39,375 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:06:44,467 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:06:49,555 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:06:59,667 | test | INFO | pool-2-thread-15 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:06:59,729 | test | INFO | pool-2-thread-15 | [task:check:414] Rebalance completed with progress: 100% in 25.6300001144 sec 2021-05-25 10:06:59,730 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:06:59,926 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:07:00,184 | test | INFO | pool-2-thread-16 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 31.4055234254 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.27504250142 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 21.2542915236 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 2.32352401142 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 1.40558226496 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 1.51 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 30.3116666667 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 30.9638493975 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.38168969483 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 1.30164497258 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:07:00,221 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:07:05,368 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 10:07:10,482 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:07:15,573 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:07:20,674 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:07:25,763 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:07:30,848 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 10:07:35,937 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 32.25 2021-05-25 10:07:41,056 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 34.5 2021-05-25 10:07:46,154 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 36.75 2021-05-25 10:07:51,266 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 10:07:56,369 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 39.0 2021-05-25 10:08:01,476 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 10:08:06,584 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 10:08:11,674 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 10:08:16,770 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:08:21,868 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:08:26,964 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:08:37,088 | test | INFO | pool-2-thread-16 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.104.247:8091 during connect (<_realsocket at 0x11 type=client open_count=1 channel=[id: 0xb71ff043, 0.0.0.0/0.0.0.0:37701] timeout=300.0>) 2021-05-25 10:08:37,154 | test | ERROR | pool-2-thread-16 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.104.247:8091/nodes/self error [Errno 111] Connection refused 2021-05-25 10:08:41,276 | test | INFO | pool-2-thread-16 | [task:check:414] Rebalance completed with progress: 100% in 101.095000029 sec 2021-05-25 10:08:41,279 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:08:41,467 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:08:47,200 | test | INFO | pool-2-thread-17 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 27.4245429243 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.35402950246 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 32.162261258 | Cluster node | | 172.23.107.129 | index, n1ql | 7.0.0-5219-enterprise | 17.9667381083 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.5648849865 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 21.9783333333 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 24.6482047173 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 20.4899073103 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 11.3014783087 | Cluster node | | 172.23.104.247 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 10:08:47,240 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:08:52,374 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 10:08:57,480 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:09:02,569 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:09:07,676 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 32.37 2021-05-25 10:09:12,780 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.74 2021-05-25 10:09:17,884 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 34.74 2021-05-25 10:09:23,005 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 37.11 2021-05-25 10:09:28,089 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 37.11 2021-05-25 10:09:33,194 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 39.47 2021-05-25 10:09:38,292 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 41.84 2021-05-25 10:09:43,381 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 41.84 2021-05-25 10:09:48,496 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 10:09:53,579 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:09:58,664 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:10:03,769 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:10:13,884 | test | INFO | pool-2-thread-17 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:10:14,005 | test | INFO | pool-2-thread-17 | [task:check:414] Rebalance completed with progress: 100% in 86.8110001087 sec 2021-05-25 10:10:14,007 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:10:14,187 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:10:19,936 | test | INFO | pool-2-thread-18 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 18.0227325756 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.86333333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 43.9316666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.6235103918 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 14.5847569207 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 58.2543042384 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 19.0886514775 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 18.9836497275 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 9.06469784341 | Cluster node | | 172.23.107.129 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 10:10:19,966 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:10:25,158 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:10:30,253 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:10:35,367 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:10:40,471 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 62.51 2021-05-25 10:10:45,595 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 65.85 2021-05-25 10:10:50,713 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 66.31 2021-05-25 10:10:55,838 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 65.43 2021-05-25 10:11:00,960 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 69.91 2021-05-25 10:11:06,104 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 75.53 2021-05-25 10:11:11,221 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.57 2021-05-25 10:11:16,322 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 76.22 2021-05-25 10:11:21,417 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 74.08 2021-05-25 10:11:26,513 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 77.38 2021-05-25 10:11:31,621 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.97 2021-05-25 10:11:36,711 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 10:11:41,803 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 10:11:46,904 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 78.94 2021-05-25 10:11:52,000 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 10:11:57,088 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:12:02,197 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:12:07,279 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:12:12,369 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:12:22,480 | test | INFO | pool-2-thread-18 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:12:22,536 | test | INFO | pool-2-thread-18 | [task:check:414] Rebalance completed with progress: 100% in 122.601999998 sec 2021-05-25 10:12:34,039 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 20 #### 2021-05-25 10:12:34,042 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:12:34,239 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:12:34,483 | test | INFO | pool-2-thread-19 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 29.9211679805 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 8.02319961334 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 13.466442226 | Cluster node | | 172.23.107.129 | eventing | 7.0.0-5219-enterprise | 7.21345355756 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.554403714 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 4.84642434545 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 27.2403046514 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 29.3829872204 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.40502341706 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.37666666667 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:12:34,513 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:12:39,677 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:12:44,763 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:12:49,918 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:12:55,105 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 60.43 2021-05-25 10:13:00,243 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 62.27 2021-05-25 10:13:05,365 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 63.89 2021-05-25 10:13:10,486 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 64.66 2021-05-25 10:13:15,594 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 65.97 2021-05-25 10:13:20,716 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 67.72 2021-05-25 10:13:25,846 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.35 2021-05-25 10:13:30,964 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.33 2021-05-25 10:13:36,075 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.33 2021-05-25 10:13:41,184 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 10:13:46,289 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 10:13:51,424 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 10:13:56,532 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.35 2021-05-25 10:14:01,644 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 10:14:06,772 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.35 2021-05-25 10:14:11,930 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.35 2021-05-25 10:14:17,345 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.34 2021-05-25 10:14:22,476 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.36 2021-05-25 10:14:27,588 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.37 2021-05-25 10:14:32,700 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.44 2021-05-25 10:14:37,809 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 68.62 2021-05-25 10:14:42,918 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.13 2021-05-25 10:14:48,035 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.13 2021-05-25 10:14:53,144 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.12 2021-05-25 10:14:58,259 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.16 2021-05-25 10:15:03,388 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 69.9 2021-05-25 10:15:08,497 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 71.63 2021-05-25 10:15:13,614 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 73.02 2021-05-25 10:15:18,746 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 73.05 2021-05-25 10:15:23,851 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 74.38 2021-05-25 10:15:28,963 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 75.32 2021-05-25 10:15:34,066 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 75.44 2021-05-25 10:15:39,187 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 77.86 2021-05-25 10:15:44,315 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:15:49,421 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:15:54,555 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:16:04,693 | test | INFO | pool-2-thread-19 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:16:05,365 | test | INFO | pool-2-thread-19 | [task:check:414] Rebalance completed with progress: 100% in 210.888000011 sec 2021-05-25 10:16:05,368 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:16:05,594 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:16:05,828 | test | INFO | pool-2-thread-20 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 70.416479176 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.50651644945 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 52.8373581321 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.727205956 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 5.18175302922 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 51.0166327789 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 70.9104848253 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.24492925118 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 6.83796242366 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:16:05,865 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:16:11,112 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 2.33 2021-05-25 10:16:16,279 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 5.72 2021-05-25 10:16:21,436 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 9.01 2021-05-25 10:16:26,585 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 12.41 2021-05-25 10:16:31,756 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 15.68 2021-05-25 10:16:36,904 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 19.12 2021-05-25 10:16:42,108 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 22.69 2021-05-25 10:16:47,280 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 25.43 2021-05-25 10:16:52,447 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 28.85 2021-05-25 10:16:57,592 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 31.77 2021-05-25 10:17:02,710 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 10:17:07,839 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 10:17:12,960 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 44.44 2021-05-25 10:17:18,075 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:17:23,168 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:17:33,295 | test | INFO | pool-2-thread-20 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:17:33,423 | test | INFO | pool-2-thread-20 | [task:check:414] Rebalance completed with progress: 100% in 87.5970001221 sec 2021-05-25 10:17:33,426 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:17:33,601 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:17:45,437 | test | INFO | pool-2-thread-21 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 90.3663455449 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.67833333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 57.2390460159 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.15714380959 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 16.0680655322 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 88.655545956 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 17.3980433659 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 6.67556965195 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:17:45,482 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:17:50,710 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 1.06 2021-05-25 10:17:55,852 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 4.82 2021-05-25 10:18:00,997 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 7.58 2021-05-25 10:18:06,157 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 11.64 2021-05-25 10:18:11,313 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 15.3 2021-05-25 10:18:16,480 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 17.31 2021-05-25 10:18:21,657 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 20.27 2021-05-25 10:18:26,836 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 26.85 2021-05-25 10:18:31,976 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 33.53 2021-05-25 10:18:37,147 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 36.86 2021-05-25 10:18:42,295 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:18:47,394 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:18:52,486 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:18:57,645 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 10:19:07,908 | test | INFO | pool-2-thread-21 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:19:07,973 | test | INFO | pool-2-thread-21 | [task:check:414] Rebalance completed with progress: 100% in 82.5380001068 sec 2021-05-25 10:19:07,976 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:19:08,198 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:19:08,428 | test | INFO | pool-2-thread-22 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 76.4712764713 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.65342178073 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 87.5506224689 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 82.8666666667 | --- OUT ---> | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.56533635878 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 9.99866662222 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 80.300150075 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 69.7726438086 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 9.55833333333 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.88343139052 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:19:08,471 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:19:13,707 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 4.84 2021-05-25 10:19:18,875 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 11.87 2021-05-25 10:19:24,039 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 18.05 2021-05-25 10:19:29,190 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 25.23 2021-05-25 10:19:34,359 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 33.36 2021-05-25 10:19:39,513 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:19:44,631 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:19:49,746 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:19:54,908 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:20:00,013 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:20:05,269 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 10:20:15,562 | test | INFO | pool-2-thread-22 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:20:15,700 | test | INFO | pool-2-thread-22 | [task:check:414] Rebalance completed with progress: 100% in 67.2750000954 sec 2021-05-25 10:20:15,700 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:20:15,908 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:20:22,246 | test | INFO | pool-2-thread-23 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 52.2942049034 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.34337238954 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 60.4689921835 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.6098677095 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 20.7283333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 55.0259170986 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 69.0131205555 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 29.1220129135 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 10.6418495952 | Cluster node | | 172.23.107.129 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:20:22,342 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:20:27,536 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 2.58 2021-05-25 10:20:32,756 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 7.8 2021-05-25 10:20:38,016 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 12.14 2021-05-25 10:20:43,163 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 16.42 2021-05-25 10:20:48,332 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 20.65 2021-05-25 10:20:53,476 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 24.4 2021-05-25 10:20:58,653 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 28.33 2021-05-25 10:21:03,818 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 10:21:08,937 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 35.42 2021-05-25 10:21:14,075 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 39.35 2021-05-25 10:21:19,211 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:21:24,334 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:21:29,451 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:21:34,568 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:21:39,671 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:21:49,788 | test | INFO | pool-2-thread-23 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:21:49,914 | test | INFO | pool-2-thread-23 | [task:check:414] Rebalance completed with progress: 100% in 87.6740000248 sec 2021-05-25 10:21:49,917 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:21:50,105 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:21:50,295 | test | INFO | pool-2-thread-24 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 71.9021349644 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.33982200593 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 65.7819416248 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 89.3634993458 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.75891906283 | --- OUT ---> | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 7.33493077742 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 70.4678411307 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.17356142044 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 2.98875058244 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:21:50,316 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:21:55,492 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:22:00,598 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 47.76 2021-05-25 10:22:05,710 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:22:10,811 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:22:15,898 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:22:26,009 | test | INFO | pool-2-thread-24 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:22:26,144 | test | INFO | pool-2-thread-24 | [task:check:414] Rebalance completed with progress: 100% in 35.8519999981 sec 2021-05-25 10:22:26,145 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:22:26,323 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:22:37,789 | test | INFO | pool-2-thread-25 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 62.7889535174 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.61993966767 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 37.4218710936 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 54.8442474041 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 11.8733333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 45.5456106139 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.359777337 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 6.14969251537 | Cluster node | | 172.23.107.126 | ['cbas'] | | | <--- IN --- | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:22:37,848 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:22:43,003 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 1.27 2021-05-25 10:22:48,101 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 11.96 2021-05-25 10:22:53,207 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 16.7 2021-05-25 10:22:58,322 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 20.27 2021-05-25 10:23:03,436 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 24.43 2021-05-25 10:23:08,552 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 28.23 2021-05-25 10:23:13,664 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 32.13 2021-05-25 10:23:18,775 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 35.58 2021-05-25 10:23:23,861 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 39.46 2021-05-25 10:23:28,993 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 10:23:34,098 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 64.54 2021-05-25 10:23:39,183 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:23:44,270 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:23:49,388 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:23:59,575 | test | INFO | pool-2-thread-25 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:23:59,653 | test | INFO | pool-2-thread-25 | [task:check:414] Rebalance completed with progress: 100% in 81.8659999371 sec 2021-05-25 10:23:59,654 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:23:59,839 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:24:00,069 | test | INFO | pool-2-thread-26 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | kv | 7.0.0-5219-enterprise | 70.4938251029 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 5.98529926496 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 89.2701788304 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 90.8475196778 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.52644190768 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 20.8079865336 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 67.8905351577 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 81.8507592046 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.1271709057 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 6.57802775651 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:24:00,096 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:24:05,279 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-05-25 10:24:10,486 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 11.72 2021-05-25 10:24:15,648 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 19.84 2021-05-25 10:24:20,812 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 27.03 2021-05-25 10:24:25,954 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 34.38 2021-05-25 10:24:31,095 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:24:36,213 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:24:41,328 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:24:46,446 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:24:51,540 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:24:56,667 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: running, progress: 85.0 2021-05-25 10:25:06,936 | test | INFO | pool-2-thread-26 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:25:07,101 | test | INFO | pool-2-thread-26 | [task:check:414] Rebalance completed with progress: 100% in 67.0369999409 sec 2021-05-25 10:25:07,102 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:25:07,342 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:25:13,141 | test | INFO | pool-2-thread-27 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 4.84658589024 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 63.9450770261 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 61.3279557348 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.36333333333 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 15.4183333333 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 59.27 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 59.895 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 20.0779987 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 5.65471726414 | Cluster node | | 172.23.107.142 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 10:25:13,187 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:25:18,401 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:25:23,490 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:25:28,582 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:25:38,709 | test | INFO | pool-2-thread-27 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:25:38,772 | test | INFO | pool-2-thread-27 | [task:check:414] Rebalance completed with progress: 100% in 25.6349999905 sec 2021-05-25 10:25:38,773 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:25:38,924 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:25:39,158 | test | INFO | pool-2-thread-28 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 2.78459458106 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.55495741738 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 16.4608436245 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 27.4733333333 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 8.34512022631 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 5.39175652928 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 32.525 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 30.9098788273 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.64841244787 | --- OUT ---> | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 1.8487394958 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:25:39,193 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:25:44,395 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 10:25:49,490 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:25:54,585 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:25:59,678 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 10:26:04,782 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 10:26:09,868 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-05-25 10:26:14,950 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 33.91 2021-05-25 10:26:20,043 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 34.57 2021-05-25 10:26:25,138 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 10:26:30,220 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 10:26:35,322 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 35.87 2021-05-25 10:26:40,423 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 37.83 2021-05-25 10:26:45,512 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 39.78 2021-05-25 10:26:50,609 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 41.74 2021-05-25 10:26:55,691 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.04 2021-05-25 10:27:00,799 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.04 2021-05-25 10:27:05,887 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 43.7 2021-05-25 10:27:10,970 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 10:27:16,058 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:27:21,157 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:27:26,255 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:27:36,368 | test | INFO | pool-2-thread-28 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:27:36,490 | test | INFO | pool-2-thread-28 | [task:check:414] Rebalance completed with progress: 100% in 117.334000111 sec 2021-05-25 10:27:36,492 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:27:36,723 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:27:42,615 | test | INFO | pool-2-thread-29 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 9.29666666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.595 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 19.5296745054 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 18.2569709495 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.03041951642 | Cluster node | | 172.23.104.247 | index, n1ql | 7.0.0-5219-enterprise | 11.0403680123 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 38.3103896537 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 31.955 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 4.66476676166 | Cluster node | | 172.23.105.183 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-05-25 10:27:42,642 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:27:47,756 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 30.0 2021-05-25 10:27:52,855 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.5 2021-05-25 10:27:57,994 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 10:28:03,095 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 31.88 2021-05-25 10:28:08,178 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 10:28:13,276 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 33.75 2021-05-25 10:28:18,390 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 35.63 2021-05-25 10:28:23,503 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 36.88 2021-05-25 10:28:28,605 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 37.5 2021-05-25 10:28:33,693 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 10:28:38,779 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 39.38 2021-05-25 10:28:43,887 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 10:28:48,976 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 41.25 2021-05-25 10:28:54,109 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 43.13 2021-05-25 10:28:59,220 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 45.0 2021-05-25 10:29:04,319 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:29:09,411 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:29:14,506 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-05-25 10:29:24,645 | test | INFO | pool-2-thread-29 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:29:24,763 | test | INFO | pool-2-thread-29 | [task:check:414] Rebalance completed with progress: 100% in 102.153000116 sec 2021-05-25 10:29:24,766 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:29:24,951 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:29:30,924 | test | INFO | pool-2-thread-30 | [table_view:display:72] Rebalance Overview +----------------+--------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+--------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.3812872904 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 11.6427511958 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 32.0277995367 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 71.3040434681 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.03464826759 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 50.5983133896 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 66.6138897685 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 22.3192560248 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 9.33478006844 | Cluster node | | 172.23.104.247 | ['eventing'] | | | <--- IN --- | +----------------+--------------+-----------------------+---------------+--------------+ 2021-05-25 10:29:30,948 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:29:36,066 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:29:41,164 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:29:46,289 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:29:51,414 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 63.89 2021-05-25 10:29:56,535 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 65.18 2021-05-25 10:30:01,647 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 66.91 2021-05-25 10:30:06,772 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 68.81 2021-05-25 10:30:11,887 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 74.01 2021-05-25 10:30:17,023 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 78.58 2021-05-25 10:30:22,128 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 77.03 2021-05-25 10:30:27,230 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 77.72 2021-05-25 10:30:32,331 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 10:30:37,417 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.98 2021-05-25 10:30:42,516 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:30:47,634 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 78.43 2021-05-25 10:30:52,759 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:30:57,861 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:31:02,963 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:31:08,053 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:31:13,171 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:31:23,312 | test | INFO | pool-2-thread-30 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:31:23,372 | test | INFO | pool-2-thread-30 | [task:check:414] Rebalance completed with progress: 100% in 112.450000048 sec 2021-05-25 10:31:34,028 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 21 #### 2021-05-25 10:31:34,029 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:31:34,216 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:31:34,430 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.7407827188 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 2.52508416947 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 19.5843471884 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 18.0463658939 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 2.84161930634 | Cluster node | | 172.23.104.247 | eventing | 7.0.0-5219-enterprise | 5.62647911736 | --- OUT ---> | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 15.8597356711 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 18.5919765329 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 11.17 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 3.09989667011 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:31:34,457 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:31:39,598 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:31:44,703 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:31:49,881 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.0 2021-05-25 10:31:55,033 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 60.57 2021-05-25 10:32:00,144 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 62.53 2021-05-25 10:32:05,250 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 64.23 2021-05-25 10:32:10,369 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 65.03 2021-05-25 10:32:15,502 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 66.77 2021-05-25 10:32:20,661 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.33 2021-05-25 10:32:25,779 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.42 2021-05-25 10:32:30,878 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.41 2021-05-25 10:32:36,009 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 68.53 2021-05-25 10:32:41,114 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 70.11 2021-05-25 10:32:46,237 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 72.68 2021-05-25 10:32:51,384 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.05 2021-05-25 10:32:56,539 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.05 2021-05-25 10:33:01,701 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.04 2021-05-25 10:33:06,825 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 73.06 2021-05-25 10:33:11,953 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 74.7 2021-05-25 10:33:17,076 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:33:22,223 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:33:27,367 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:33:32,480 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 79.99 2021-05-25 10:33:42,615 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:33:42,743 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 128.319000006 sec 2021-05-25 10:33:42,746 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:33:42,934 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:33:43,141 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 16.246937449 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.676677944632 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 45.2609123181 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 46.4325594573 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.523324611256 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 49.1649443296 | --- OUT ---> | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 71.9795329922 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.72 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 0.961666666667 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:33:43,174 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:33:48,335 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 2.9 2021-05-25 10:33:53,535 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 6.75 2021-05-25 10:33:58,729 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 10.25 2021-05-25 10:34:03,903 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 13.08 2021-05-25 10:34:09,069 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 16.48 2021-05-25 10:34:14,250 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 19.86 2021-05-25 10:34:19,461 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 22.57 2021-05-25 10:34:24,650 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 25.62 2021-05-25 10:34:29,798 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 28.82 2021-05-25 10:34:34,990 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 32.04 2021-05-25 10:34:40,128 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 10:34:45,250 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-05-25 10:34:50,378 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:34:55,484 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:35:00,608 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:35:10,780 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:35:10,963 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 87.8250000477 sec 2021-05-25 10:35:10,964 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:35:11,224 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:35:24,046 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 1.84672822427 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.47 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 58.0466666667 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 88.9603679877 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 5.10482983901 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 84.3478115937 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 1.97506583553 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 0.731654472425 | Cluster node | | 172.23.104.247 | ['kv'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:35:24,092 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:35:29,309 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.84 2021-05-25 10:35:34,568 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 2.62 2021-05-25 10:35:39,782 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 7.08 2021-05-25 10:35:45,032 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.04 2021-05-25 10:35:50,275 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 10.39 2021-05-25 10:35:55,460 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.35 2021-05-25 10:36:00,598 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 15.5 2021-05-25 10:36:05,805 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 17.53 2021-05-25 10:36:11,032 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 21.69 2021-05-25 10:36:16,282 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 25.14 2021-05-25 10:36:21,611 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 29.76 2021-05-25 10:36:26,796 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 34.59 2021-05-25 10:36:31,979 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 38.38 2021-05-25 10:36:37,155 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:36:42,266 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:36:47,413 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:36:57,572 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:36:57,651 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 93.6089999676 sec 2021-05-25 10:36:57,654 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:36:57,903 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.105.1 2021-05-25 10:36:58,232 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 14.324761254 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 10.6518441974 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 76.4320594657 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 83.8041902095 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 24.2750483301 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 76.743720938 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 68.9571840469 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 77.3440885304 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 13.9568992817 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 11.0151835864 | Cluster node | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:36:58,275 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:37:03,450 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 5.47 2021-05-25 10:37:08,641 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 13.12 2021-05-25 10:37:13,884 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 19.76 2021-05-25 10:37:19,073 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 26.48 2021-05-25 10:37:24,305 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 32.42 2021-05-25 10:37:29,466 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 37.42 2021-05-25 10:37:37,622 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:37:42,785 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:37:47,924 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:37:53,049 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:37:58,167 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:38:03,296 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:38:08,446 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 72.35 2021-05-25 10:38:13,572 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 77.06 2021-05-25 10:38:18,690 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 79.02 2021-05-25 10:38:28,851 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:38:29,056 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 90.8279998302 sec 2021-05-25 10:38:29,058 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:38:29,315 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:38:38,684 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 14.7624047869 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 7.9684661411 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 62.6014634065 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 54.6109101818 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 10.0518341972 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 60.755358512 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 51.563281224 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 15.6011466284 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 8.77652039133 | Cluster node | | 172.23.105.1 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:38:38,733 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:38:43,858 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:38:49,013 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 2.87 2021-05-25 10:38:54,214 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 9.56 2021-05-25 10:38:59,424 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 14.77 2021-05-25 10:39:04,661 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 20.11 2021-05-25 10:39:09,851 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 25.04 2021-05-25 10:39:15,038 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 30.43 2021-05-25 10:39:20,242 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 35.07 2021-05-25 10:39:25,415 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:39:30,572 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:39:35,740 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:39:40,885 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:39:46,020 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:39:51,140 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:39:56,257 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 71.69 2021-05-25 10:40:01,401 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 75.06 2021-05-25 10:40:06,526 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 77.66 2021-05-25 10:40:11,664 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 77.27 2021-05-25 10:40:16,796 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 77.79 2021-05-25 10:40:21,918 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 79.35 2021-05-25 10:40:32,098 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:40:32,276 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 113.59800005 sec 2021-05-25 10:40:32,279 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:40:32,551 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:40:32,868 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 15.0221674056 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 3.85493575107 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 29.5833333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 62.657911403 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 7.66670549666 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 56.1704113608 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 34.476149205 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 4.32326127898 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 6.8378486321 | --- OUT ---> | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:40:32,907 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:40:38,043 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 48.86 2021-05-25 10:40:43,173 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:40:51,289 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-05-25 10:41:01,461 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-05-25 10:41:01,601 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 28.7369999886 sec 2021-05-25 10:41:01,604 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:41:01,867 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:41:14,188 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+---------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 11.5116666667 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 10.2583333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 30.1311644806 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 22.9166666667 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 45.2258538377 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 22.9518595179 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 28.6421440357 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 10.2883333333 | Cluster node | | 172.23.107.131 | ['cbas'] | | | <--- IN --- | | 172.23.105.137 | ['kv'] | | | <--- IN --- | +----------------+-------------+-----------------------+---------------+--------------+ 2021-05-25 10:41:14,234 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-05-25 10:41:19,397 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.34 2021-05-25 10:41:24,573 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 11.27 2021-05-25 10:41:32,729 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 20.78 2021-05-25 10:41:37,862 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 25.2 2021-05-25 10:41:42,996 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 28.76 2021-05-25 10:41:48,127 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 32.72 2021-05-25 10:41:53,250 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 36.65 2021-05-25 10:41:58,411 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-05-25 10:42:03,539 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-05-25 10:42:08,653 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 68.07 2021-05-25 10:42:13,766 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:42:18,869 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:42:23,986 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 70.0 2021-05-25 10:42:29,121 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 72.32 2021-05-25 10:42:34,263 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 78.62 2021-05-25 10:42:39,392 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:42:44,523 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:42:49,667 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:42:54,802 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:42:59,927 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:05,042 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:10,154 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:15,288 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:20,410 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:25,533 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:30,644 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:35,750 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:40,858 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:45,944 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:51,035 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:43:56,125 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:01,227 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:06,319 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:11,424 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:16,516 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:21,612 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:26,720 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:31,819 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:36,927 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:42,035 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:47,128 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:47,526 | test | INFO | pool-2-thread-7 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.104.247: 2021-05-25 10:44:47,528 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528162L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.162Z', u'text': u'Analytics collection hotels.hotel_with_low_ratings rebalanced. Rebalance progress now is 0.99. Remaining analytics collections: 0/17'} 2021-05-25 10:44:47,529 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528065L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.065Z', u'text': u'Analytics collection hotels.hotel_with_good_ratings rebalanced. Rebalance progress now is 0.9517713628853349. Remaining analytics collections: 1/17'} 2021-05-25 10:44:47,529 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527929L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.929Z', u'text': u'Analytics collection airlines.agent_5_user_bookings rebalanced. Rebalance progress now is 0.9035427257706699. Remaining analytics collections: 2/17'} 2021-05-25 10:44:47,530 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527809L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.809Z', u'text': u'Analytics collection airlines.agent_4_user_bookings rebalanced. Rebalance progress now is 0.815280796283506. Remaining analytics collections: 3/17'} 2021-05-25 10:44:47,530 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527680L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.680Z', u'text': u'Analytics collection airlines.agent_3_user_bookings rebalanced. Rebalance progress now is 0.727011545098001. Remaining analytics collections: 4/17'} 2021-05-25 10:44:47,532 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527473L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.473Z', u'text': u'Analytics collection airlines.agent_2_user_bookings rebalanced. Rebalance progress now is 0.6387443531401544. Remaining analytics collections: 5/17'} 2021-05-25 10:44:47,532 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527309L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.309Z', u'text': u'Analytics collection airlines.agent_1_user_bookings rebalanced. Rebalance progress now is 0.5504879149267464. Remaining analytics collections: 6/17'} 2021-05-25 10:44:47,532 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527202L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.202Z', u'text': u'Analytics collection agent_5.user_senior rebalanced. Rebalance progress now is 0.4822514405441486. Remaining analytics collections: 7/17'} 2021-05-25 10:44:47,533 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527083L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.083Z', u'text': u'Analytics collection agent_5.user_middle_age rebalanced. Rebalance progress now is 0.4340275320263288. Remaining analytics collections: 8/17'} 2021-05-25 10:44:47,533 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964526941L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:06.941Z', u'text': u'Analytics collection agent_4.user_senior rebalanced. Rebalance progress now is 0.385802174422379. Remaining analytics collections: 9/17'} 2021-05-25 10:44:47,536 | test | ERROR | pool-2-thread-7 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-05-25 10:44:47,538 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:44:47,739 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:44:47,903 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 1.77163713938 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 1.3250220837 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 11.5337177906 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 29.7401796577 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 6.58269131571 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 15.7564040599 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 14.685 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 28.5916666667 | --- OUT ---> | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.49490841819 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 0.573333333333 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:44:47,927 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:53,002 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:44:58,108 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:03,209 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:08,292 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:13,385 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:18,483 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:23,605 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:28,693 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:33,780 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:38,865 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:43,967 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:49,069 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:54,165 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:45:59,266 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:04,368 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:09,451 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:14,552 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:19,647 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:24,743 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:29,825 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:34,910 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:40,009 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:45,114 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:50,221 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:55,334 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:46:55,849 | test | INFO | pool-2-thread-8 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.104.247: 2021-05-25 10:46:55,849 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964687935L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:44:47.935Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:46:55,851 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528162L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.162Z', u'text': u'Analytics collection hotels.hotel_with_low_ratings rebalanced. Rebalance progress now is 0.99. Remaining analytics collections: 0/17'} 2021-05-25 10:46:55,851 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528065L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.065Z', u'text': u'Analytics collection hotels.hotel_with_good_ratings rebalanced. Rebalance progress now is 0.9517713628853349. Remaining analytics collections: 1/17'} 2021-05-25 10:46:55,851 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527929L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.929Z', u'text': u'Analytics collection airlines.agent_5_user_bookings rebalanced. Rebalance progress now is 0.9035427257706699. Remaining analytics collections: 2/17'} 2021-05-25 10:46:55,851 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527809L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.809Z', u'text': u'Analytics collection airlines.agent_4_user_bookings rebalanced. Rebalance progress now is 0.815280796283506. Remaining analytics collections: 3/17'} 2021-05-25 10:46:55,852 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527680L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.680Z', u'text': u'Analytics collection airlines.agent_3_user_bookings rebalanced. Rebalance progress now is 0.727011545098001. Remaining analytics collections: 4/17'} 2021-05-25 10:46:55,854 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527473L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.473Z', u'text': u'Analytics collection airlines.agent_2_user_bookings rebalanced. Rebalance progress now is 0.6387443531401544. Remaining analytics collections: 5/17'} 2021-05-25 10:46:55,854 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527309L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.309Z', u'text': u'Analytics collection airlines.agent_1_user_bookings rebalanced. Rebalance progress now is 0.5504879149267464. Remaining analytics collections: 6/17'} 2021-05-25 10:46:55,855 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527202L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.202Z', u'text': u'Analytics collection agent_5.user_senior rebalanced. Rebalance progress now is 0.4822514405441486. Remaining analytics collections: 7/17'} 2021-05-25 10:46:55,855 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527083L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.083Z', u'text': u'Analytics collection agent_5.user_middle_age rebalanced. Rebalance progress now is 0.4340275320263288. Remaining analytics collections: 8/17'} 2021-05-25 10:46:55,855 | test | ERROR | pool-2-thread-8 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-05-25 10:46:55,857 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-05-25 10:46:56,105 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:46:56,286 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 5.39315356155 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.468333333333 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 29.0104835081 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 14.5578480717 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.11353901028 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 27.7000634327 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 13.0481158647 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 36.095 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 6.69033451673 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 10.784820253 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:46:56,313 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:01,398 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:06,516 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:11,601 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:16,691 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:21,806 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:26,914 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:32,033 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:37,121 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:42,221 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:47,323 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:52,417 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:47:57,542 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:02,648 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:07,746 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:12,834 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:17,931 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:23,035 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:28,122 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:33,207 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:38,292 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:43,381 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:48,490 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:53,582 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:48:58,678 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:03,779 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:04,157 | test | INFO | pool-2-thread-9 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.104.247: 2021-05-25 10:49:04,158 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964816373L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:46:56.373Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:49:04,160 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964687935L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:44:47.935Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:49:04,161 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528162L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.162Z', u'text': u'Analytics collection hotels.hotel_with_low_ratings rebalanced. Rebalance progress now is 0.99. Remaining analytics collections: 0/17'} 2021-05-25 10:49:04,161 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528065L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.065Z', u'text': u'Analytics collection hotels.hotel_with_good_ratings rebalanced. Rebalance progress now is 0.9517713628853349. Remaining analytics collections: 1/17'} 2021-05-25 10:49:04,161 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527929L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.929Z', u'text': u'Analytics collection airlines.agent_5_user_bookings rebalanced. Rebalance progress now is 0.9035427257706699. Remaining analytics collections: 2/17'} 2021-05-25 10:49:04,161 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527809L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.809Z', u'text': u'Analytics collection airlines.agent_4_user_bookings rebalanced. Rebalance progress now is 0.815280796283506. Remaining analytics collections: 3/17'} 2021-05-25 10:49:04,163 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527680L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.680Z', u'text': u'Analytics collection airlines.agent_3_user_bookings rebalanced. Rebalance progress now is 0.727011545098001. Remaining analytics collections: 4/17'} 2021-05-25 10:49:04,164 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527473L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.473Z', u'text': u'Analytics collection airlines.agent_2_user_bookings rebalanced. Rebalance progress now is 0.6387443531401544. Remaining analytics collections: 5/17'} 2021-05-25 10:49:04,164 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527309L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.309Z', u'text': u'Analytics collection airlines.agent_1_user_bookings rebalanced. Rebalance progress now is 0.5504879149267464. Remaining analytics collections: 6/17'} 2021-05-25 10:49:04,164 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527202L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.202Z', u'text': u'Analytics collection agent_5.user_senior rebalanced. Rebalance progress now is 0.4822514405441486. Remaining analytics collections: 7/17'} 2021-05-25 10:49:04,165 | test | ERROR | pool-2-thread-9 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-05-25 10:49:04,167 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-05-25 10:49:04,357 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 2021-05-25 10:49:04,513 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 5.1064113461 | Cluster node | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.620010333506 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 26.9533333333 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 40.9856830947 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 0.753345889098 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 52.2142035701 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 42.7621491901 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 32.8894518425 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 5.18482717243 | --- OUT ---> | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 0.696724727061 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:49:04,539 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:09,607 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:14,711 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:19,801 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:24,895 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:29,994 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:35,081 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:40,184 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:45,273 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:50,375 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:49:55,467 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:00,585 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:05,710 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:10,799 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:15,895 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:21,009 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:26,105 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:31,213 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:36,296 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:41,388 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:46,480 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:51,585 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:50:56,698 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:51:01,802 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:51:06,901 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:51:11,996 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 79.8 2021-05-25 10:51:12,328 | test | INFO | pool-2-thread-10 | [rest_client:print_UI_logs:2694] Latest logs from UI on 172.23.104.247: 2021-05-25 10:51:12,328 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964944642L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:49:04.642Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:51:12,329 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964816373L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:46:56.373Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:51:12,331 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.104.247', u'tstamp': 1621964687935L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:44:47.935Z', u'text': u'Not rebalancing because rebalance is already in progress.~n'} 2021-05-25 10:51:12,331 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528162L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.162Z', u'text': u'Analytics collection hotels.hotel_with_low_ratings rebalanced. Rebalance progress now is 0.99. Remaining analytics collections: 0/17'} 2021-05-25 10:51:12,332 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964528065L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:08.065Z', u'text': u'Analytics collection hotels.hotel_with_good_ratings rebalanced. Rebalance progress now is 0.9517713628853349. Remaining analytics collections: 1/17'} 2021-05-25 10:51:12,332 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527929L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.929Z', u'text': u'Analytics collection airlines.agent_5_user_bookings rebalanced. Rebalance progress now is 0.9035427257706699. Remaining analytics collections: 2/17'} 2021-05-25 10:51:12,332 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527809L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.809Z', u'text': u'Analytics collection airlines.agent_4_user_bookings rebalanced. Rebalance progress now is 0.815280796283506. Remaining analytics collections: 3/17'} 2021-05-25 10:51:12,334 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527680L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.680Z', u'text': u'Analytics collection airlines.agent_3_user_bookings rebalanced. Rebalance progress now is 0.727011545098001. Remaining analytics collections: 4/17'} 2021-05-25 10:51:12,334 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527473L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.473Z', u'text': u'Analytics collection airlines.agent_2_user_bookings rebalanced. Rebalance progress now is 0.6387443531401544. Remaining analytics collections: 5/17'} 2021-05-25 10:51:12,335 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2696] {u'code': 0, u'module': u'analytics', u'type': u'info', u'node': u'ns_1@172.23.107.126', u'tstamp': 1621964527309L, u'shortText': u'message', u'serverTime': u'2021-05-25T10:42:07.309Z', u'text': u'Analytics collection airlines.agent_1_user_bookings rebalanced. Rebalance progress now is 0.5504879149267464. Remaining analytics collections: 6/17'} 2021-05-25 10:51:12,335 | test | ERROR | pool-2-thread-10 | [task:call:267] Rebalance Failed: seems like rebalance hangs. please check logs! 2021-05-25 10:51:12,336 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-05-25 10:51:12,552 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.104.247 ERROR2021-05-25 10:51:12,645 | test | INFO | MainThread | [app_basetest:_AppBase__print_step:90] ############################################################ 1. Closing SDK client connections ############################################################ 2021-05-25 10:51:12,733 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+-------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+-------------+-----------------------+----------------+--------------+ | 172.23.107.142 | index, n1ql | 7.0.0-5219-enterprise | 1.43166666667 | --- OUT ---> | | 172.23.106.116 | backup | 7.0.0-5219-enterprise | 0.8099730009 | Cluster node | | 172.23.107.127 | eventing | 7.0.0-5219-enterprise | 30.7823072564 | Cluster node | | 172.23.107.129 | kv | 7.0.0-5219-enterprise | 36.5872764546 | Cluster node | | 172.23.107.126 | cbas | 7.0.0-5219-enterprise | 4.61151294957 | Cluster node | | 172.23.104.247 | kv | 7.0.0-5219-enterprise | 54.8372603834 | Cluster node | | 172.23.105.137 | kv | 7.0.0-5219-enterprise | 57.2135737858 | Cluster node | | 172.23.105.1 | kv | 7.0.0-5219-enterprise | 37.4739578993 | Cluster node | | 172.23.105.183 | index, n1ql | 7.0.0-5219-enterprise | 2.89 | Cluster node | | 172.23.107.131 | cbas | 7.0.0-5219-enterprise | 0.676666666667 | Cluster node | +----------------+-------------+-----------------------+----------------+--------------+ 2021-05-25 10:51:12,759 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 79.8 ====================================================================== ERROR: run_app (bucket_collections.travel_sample_app.TravelSampleApp) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/travel_sample_app.py", line 152, in run_app cluster_scenario.run(op_type, File "pytests/bucket_collections/app/scenarios/cluster.py", line 190, in run During the test, Remote Connections: 221, Disconnections: 221 Cluster.scenarios["scenario_%s" % op_type](self, kwargs) File "pytests/bucket_collections/app/scenarios/cluster.py", line 139, in scenario_rebalance_swap self.cluster.master = self.spare_nodes[0] IndexError: index out of range: 0 ---------------------------------------------------------------------- SDK Connections: 0, Disconnections: 0 Ran 1 test in 20229.985s FAILED (errors=1) summary so far suite bucket_collections.travel_sample_app.TravelSampleApp , pass 0 , fail 1 failures so far... bucket_collections.travel_sample_app.TravelSampleApp.run_app testrunner logs, diags and results are available under /data/workspace/temp_collections_e2e_app/logs/testrunner-21-May-25_05-14-12/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 5h 37m 47s 2 actionable tasks: 2 executed + '[' false == true ']' Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 1.1.6 -P client_version=default' [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Finished: UNSTABLE