Started by user Ashwin G [EnvInject] - Loading node environment variables. Building remotely on slv-s83002 (P0_win_upgrade jython_slave) in workspace /data/workspace/temp_centos7_4node_jython Wiping out workspace first. Cloning the remote Git repository Cloning repository https://github.com/couchbaselabs/TAF.git > /usr/bin/git init /data/workspace/temp_centos7_4node_jython # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git --version # timeout=10 > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 > /usr/bin/git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 > /usr/bin/git config remote.origin.url https://github.com/couchbaselabs/TAF.git # timeout=10 Fetching upstream changes from https://github.com/couchbaselabs/TAF.git > /usr/bin/git fetch --tags --progress https://github.com/couchbaselabs/TAF.git +refs/heads/*:refs/remotes/origin/* > /usr/bin/git rev-parse origin/master^{commit} # timeout=10 Checking out Revision 0cfc2922608d5d570384d3281ccbb4c1b67bf993 (origin/master) > /usr/bin/git config core.sparsecheckout # timeout=10 > /usr/bin/git checkout -f 0cfc2922608d5d570384d3281ccbb4c1b67bf993 > /usr/bin/git rev-list 0cfc2922608d5d570384d3281ccbb4c1b67bf993 # timeout=10 > /usr/bin/git tag -a -f -m Jenkins Build #1918 jenkins-temp_centos7_4node_jython-1918 # timeout=10 No emails were triggered. [temp_centos7_4node_jython] $ /bin/sh -xe /tmp/jenkins2135486334666938537.sh + exit 0 [temp_centos7_4node_jython] $ /bin/bash -x /tmp/jenkins2645332104424741811.sh + jython_bin=/opt/jython/bin + jython_path=/opt/jython/bin/jython + ini_file=/tmp/oel6-4node-rebalance_in_jython.ini + git clone https://github.com/gradle/guides.git Cloning into 'guides'... + echo ' [global] #username:root #password:couchbase username:Administrator password:Membase123 port:8091 n1ql_port:8093 index_port:9102 # data_path:/data/dta # index_path:/data/idx # cbas_path:['\''/data/d1'\'','\''/data/d2'\'','\''/data/d3'\'','\''/data/d4'\'','\''/data/d5'\'','\''/data/d6'\''] [servers] #1:_1 #2:_2 #3:_3 #4:_4 #5:_5 #6:_6 #1:w1 #2:w2 #3:w3 #4:w4 #5:w5 #6:w6 #7:w7 #8:w8 #9:w9 #10:w10 #11:w11 #12:w12 #13:w13 #14:w14 #15:w15 1:w6 2:w7 3:w8 4:w9 5:w10 6:w11 7:w12 8:w13 9:w14 10:w15 [_1] ip:172.23.105.155 services: kv [_2] ip:172.23.105.211 [_3] ip:172.23.105.212 [_4] ip:172.23.105.213 [_5] ip:172.23.105.244 [_6] ip:172.23.105.245 [w1] ip:172.23.120.144 [w2] ip:172.23.120.113 [w3] ip:172.23.121.81 [w4] ip:172.23.120.100 [w5] ip:172.23.120.117 [w6] ip:172.23.136.105 [w7] ip:172.23.136.106 [w8] ip:172.23.136.107 [w9] ip:172.23.136.108 [w10] ip:172.23.136.110 [w11] ip:172.23.136.112 [w12] ip:172.23.136.113 [w13] ip:172.23.136.114 [w14] ip:172.23.136.115 [w15] ip:172.23.138.127 [membase] rest_username:Administrator rest_password:password ' + echo 'bucket_collections.travel_sample_app.TravelSampleApp: run_app,skip_setup_cleanup=True,playbook=rebalance,iteration=1000 ' + rm -rf 'logs/*' jars + '[' '' '!=' '' ']' + sdk_client_params='-P transaction_version=1.1.5 -P client_version=default' + /opt/jython/bin/pip install -r requirements.txt Requirement already satisfied (use --upgrade to upgrade): futures==3.3.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 1)) Requirement already satisfied (use --upgrade to upgrade): requests==2.24.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 2)) Requirement already satisfied (use --upgrade to upgrade): urllib3==1.25.10 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 3)) Requirement already satisfied (use --upgrade to upgrade): ruamel.yaml==0.16.12 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 4)) Requirement already satisfied (use --upgrade to upgrade): six==1.15.0 in /opt/jython/Lib/site-packages (from -r requirements.txt (line 5)) Cleaning up... + '[' false == true ']' + rm -f /tmp/test.pcap + '[' false == true ']' + '[' '-p rerun=False,skip_cleanup=True' == '' ']' + test_params='-p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017' + echo ' activities: - type: "cluster" op_type: "rebalance_in" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "cbas" - type: "cluster" op_type: "rebalance_in" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_swap" service: "n1ql:index" - type: "cluster" op_type: "rebalance_in" service: "n1ql:index" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "n1ql:index" - type: "cluster" op_type: "rebalance_in" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_out" nodes: 1 service: "kv" - type: "cluster" op_type: "rebalance_in" service: "cbas" - type: "cluster" op_type: "rebalance_out" nodes: 2 service: "kv,kv" ' + guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P transaction_version=1.1.5 -P client_version=default -P 'args=-i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest' Starting a Gradle Daemon, 15 busy Daemons could not be reused, use --status for details > Configure project : Executing 'gradle clean' Using Transaction_client :: 1.1.5 Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/ssh.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar scripts/eagles_all_around.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main scripts/install.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources testrunner.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest Running: /opt/jython/bin/jython -J-cp /data/workspace/temp_centos7_4node_jython/build/classes/java/main:/data/workspace/temp_centos7_4node_jython/build/resources/main:/root/.gradle/caches/modules-2/files-2.1/com.jcraft/jsch/0.1.54/da3584329a263616e277e15462b387addd1b208d/jsch-0.1.54.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-slf4j-impl/2.11.1/4b41b53a3a2d299ce381a69d165381ca19f62912/log4j-slf4j-impl-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/couchbase-transactions/1.1.5/babf172722b875c03d9d1f994f03d6abbbd9c270/couchbase-transactions-1.1.5.jar:/root/.gradle/caches/modules-2/files-2.1/org.slf4j/slf4j-api/1.7.25/da76ca59f6a57ee3102f8f9bd9cee742973efa8a/slf4j-api-1.7.25.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-core/2.11.1/592a48674c926b01a9a747c7831bcd82a9e6d6e4/log4j-core-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.apache.logging.log4j/log4j-api/2.11.1/268f0fe4df3eefe052b57c87ec48517d64fb2a10/log4j-api-2.11.1.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/java-client/3.1.2/d6b17914fb407cd462812e15e51dd3320780f87e/java-client-3.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/com.couchbase.client/core-io/2.1.2/545cb2cd16d56082b945b26ea77da5f5b87c736/core-io-2.1.2.jar:/root/.gradle/caches/modules-2/files-2.1/io.projectreactor/reactor-core/3.4.1/a443a39c29a2cc7b53c3d165afad456c78e5db2b/reactor-core-3.4.1.jar:/root/.gradle/caches/modules-2/files-2.1/org.reactivestreams/reactive-streams/1.0.3/d9fb7a7926ffa635b3dcaa5049fb2bfa25b3e7d0/reactive-streams-1.0.3.jar:build/classes/java/main:src/main/resources scripts/rerun_jobs.py -i /tmp/oel6-4node-rebalance_in_jython.ini -c /tmp/test_run.conf -p rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -m rest > Task :compileJava Note: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. > Task :testrunner Filename: /tmp/test_run.conf Prefix: bucket_collections.travel_sample_app.TravelSampleApp Global Test input params: {'cluster_name': 'oel6-4node-rebalance_in_jython', 'conf_file': '/tmp/test_run.conf', 'ini': '/tmp/oel6-4node-rebalance_in_jython.ini', 'num_nodes': 10, 'rerun': 'False', 'skip_cleanup': 'True', 'spec': 'test_run', 'upgrade_version': '7.0.0-5017'} Logs will be stored at /data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Apr-27_22-08-27/test_1 guides/gradlew --refresh-dependencies testrunner -P jython=/opt/jython/bin/jython -P 'args=-i /tmp/oel6-4node-rebalance_in_jython.ini rerun=False,skip_cleanup=True,upgrade_version=7.0.0-5017 -t bucket_collections.travel_sample_app.TravelSampleApp.run_app,skip_setup_cleanup=True,playbook=rebalance,iteration=1000' Test Input params: {'cluster_name': 'oel6-4node-rebalance_in_jython', 'ini': '/tmp/oel6-4node-rebalance_in_jython.ini', 'conf_file': '/tmp/test_run.conf', 'upgrade_version': '7.0.0-5017', 'spec': 'test_run', 'rerun': 'False', 'num_nodes': 10, 'logs_folder': '/data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Apr-27_22-08-27/test_1', 'skip_cleanup': 'True', 'iteration': '1000', 'case_number': 1, 'skip_setup_cleanup': 'True', 'playbook': 'rebalance'} run_app (bucket_collections.travel_sample_app.TravelSampleApp) ... 2021-04-27 22:08:39,479 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-04-27 22:08:40,499 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= AppBase setup started for test #1 run_app ========= 2021-04-27 22:08:40,684 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= AppBase setup complete for test #1 run_app ========= 2021-04-27 22:08:40,687 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup started for test #1 run_app ========= 2021-04-27 22:08:40,822 | test | INFO | MainThread | [basetestcase:log_setup_status:542] ========= TravelSampleApp setup complete for test #1 run_app ========= 2021-04-27 22:08:41,071 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:08:41,075 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 1 #### 2021-04-27 22:08:41,076 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:08:41,325 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:08:50,362 | test | INFO | pool-2-thread-1 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 24.075 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 43.78880176 | Cluster node | | 172.23.136.107 | index, n1ql | 7.0.0-5017-enterprise | 12.2294904379 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 0.354154861505 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 0.208331597237 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 0.130832243065 | Cluster node | | 172.23.136.110 | cbas | 7.0.0-5017-enterprise | 0.104167534729 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 43.6461970516 | Cluster node | | 172.23.136.108 | ['kv'] | | | <--- IN --- | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-27 22:08:50,404 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:08:56,049 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:09:01,153 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:09:06,239 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:09:11,325 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:09:16,424 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:21,513 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:26,601 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:31,706 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:36,792 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:41,881 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:46,974 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:52,059 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:09:57,153 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:02,253 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:07,341 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:12,434 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:17,532 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:22,618 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:27,707 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.92 2021-04-27 22:10:32,802 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.99 2021-04-27 22:10:37,891 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 2.99 2021-04-27 22:10:42,992 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 4.55 2021-04-27 22:10:48,085 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:10:53,176 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:10:58,259 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:03,385 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:08,496 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:13,609 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:18,693 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:23,778 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:28,891 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:33,974 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:39,055 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:44,141 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:49,211 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:54,286 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:11:59,375 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:04,453 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:09,549 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:14,635 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:19,732 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:24,812 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:29,901 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:35,003 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:40,085 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.08 2021-04-27 22:12:45,180 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.21 2021-04-27 22:12:50,266 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.21 2021-04-27 22:12:55,368 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.21 2021-04-27 22:13:00,444 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 5.21 2021-04-27 22:13:05,517 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:10,592 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:15,683 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:20,767 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:25,855 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:30,937 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:36,013 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:41,092 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:46,180 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:51,266 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:13:56,345 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:01,434 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:06,526 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:11,618 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:16,713 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:21,842 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:26,917 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:31,994 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:37,085 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:42,167 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:47,246 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:52,322 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:14:57,385 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:15:02,453 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.56 2021-04-27 22:15:07,530 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.64 2021-04-27 22:15:12,614 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.64 2021-04-27 22:15:17,690 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.67 2021-04-27 22:15:22,756 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 7.67 2021-04-27 22:15:27,828 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 9.84 2021-04-27 22:15:32,921 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.16 2021-04-27 22:15:38,010 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:15:43,089 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:15:48,164 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:15:53,250 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:15:58,328 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:16:03,408 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:16:08,486 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:16:13,572 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 10.99 2021-04-27 22:16:18,654 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 11.07 2021-04-27 22:16:23,726 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 12.03 2021-04-27 22:16:28,802 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 14.1 2021-04-27 22:16:33,880 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 16.37 2021-04-27 22:16:38,953 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 18.64 2021-04-27 22:16:44,025 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 21.09 2021-04-27 22:16:49,109 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 22.55 2021-04-27 22:16:54,190 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 24.32 2021-04-27 22:16:59,286 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 25.59 2021-04-27 22:17:04,368 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 26.41 2021-04-27 22:17:09,443 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 27.41 2021-04-27 22:17:14,532 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 28.1 2021-04-27 22:17:19,615 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 28.84 2021-04-27 22:17:24,690 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 29.51 2021-04-27 22:17:29,773 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 30.05 2021-04-27 22:17:34,841 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 30.73 2021-04-27 22:17:39,970 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.26 2021-04-27 22:17:45,052 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 31.96 2021-04-27 22:17:50,142 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 32.45 2021-04-27 22:17:55,224 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 32.84 2021-04-27 22:18:00,305 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: running, progress: 33.33 2021-04-27 22:18:10,404 | test | INFO | pool-2-thread-1 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-27 22:18:10,453 | test | INFO | pool-2-thread-1 | [task:check:414] Rebalance completed with progress: 100% in 560.092000008 sec 2021-04-27 22:18:10,453 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-27 22:18:10,598 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:18:10,783 | test | INFO | pool-2-thread-2 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 6.02744977125 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 93.32066635 | Cluster node | | 172.23.136.107 | index, n1ql | 7.0.0-5017-enterprise | 18.165 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 43.4738403587 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 88.7506458441 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 1.90083333333 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 3.20327994533 | Cluster node | | 172.23.136.110 | cbas | 7.0.0-5017-enterprise | 23.8143651197 | --- OUT ---> | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 93.4643300584 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:18:10,813 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:18:15,920 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: running, progress: 66.67 2021-04-27 22:18:26,017 | test | INFO | pool-2-thread-2 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-27 22:18:26,727 | test | INFO | pool-2-thread-2 | [task:check:414] Rebalance completed with progress: 100% in 15.9479999542 sec 2021-04-27 22:18:26,730 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:18:26,859 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:18:36,851 | test | INFO | pool-2-thread-3 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 7.33463326834 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 93.0348632429 | Cluster node | | 172.23.136.107 | index, n1ql | 7.0.0-5017-enterprise | 21.6145269088 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 22.9826915224 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 89.5567536937 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 3.47910868152 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 3.89259731493 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 92.3433333333 | Cluster node | | 172.23.136.110 | ['kv'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:18:36,877 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:18:41,953 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:18:47,028 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 1.13 2021-04-27 22:18:52,141 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 5.12 2021-04-27 22:18:57,220 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 7.73 2021-04-27 22:19:02,299 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:07,375 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:12,450 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:17,526 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:22,599 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:27,678 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:32,752 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.77 2021-04-27 22:19:37,836 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 8.86 2021-04-27 22:19:42,931 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 12.59 2021-04-27 22:19:48,049 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 16.15 2021-04-27 22:19:53,138 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 19.62 2021-04-27 22:19:58,223 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 23.01 2021-04-27 22:20:03,323 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 26.13 2021-04-27 22:20:08,441 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 28.91 2021-04-27 22:20:13,525 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 31.86 2021-04-27 22:20:18,609 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 34.64 2021-04-27 22:20:23,686 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 37.24 2021-04-27 22:20:28,763 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 39.85 2021-04-27 22:20:33,844 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 42.19 2021-04-27 22:20:38,911 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 43.14 2021-04-27 22:20:43,989 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: running, progress: 55.56 2021-04-27 22:20:54,081 | test | INFO | pool-2-thread-3 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-27 22:20:54,124 | test | INFO | pool-2-thread-3 | [task:check:414] Rebalance completed with progress: 100% in 137.273000002 sec 2021-04-27 22:20:54,125 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-04-27 22:20:54,262 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:20:59,440 | test | INFO | pool-2-thread-4 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 5.9257839518 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 91.8649322078 | Cluster node | | 172.23.136.107 | index, n1ql | 7.0.0-5017-enterprise | 22.2387039784 | --- OUT ---> | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 31.3414054883 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 93.4256689751 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 3.18493665181 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 3.92323525245 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 92.5649380412 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 90.4830954107 | Cluster node | | 172.23.136.113 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-04-27 22:20:59,461 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:21:04,539 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 40.0 2021-04-27 22:21:09,608 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:14,680 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:19,773 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:24,895 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:29,971 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:35,056 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 41.5 2021-04-27 22:21:40,148 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.37 2021-04-27 22:21:45,224 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 42.37 2021-04-27 22:21:50,298 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 43.16 2021-04-27 22:21:55,401 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 43.55 2021-04-27 22:22:00,484 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 43.55 2021-04-27 22:22:05,579 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 43.95 2021-04-27 22:22:10,671 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 44.74 2021-04-27 22:22:15,752 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 44.74 2021-04-27 22:22:20,834 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 45.53 2021-04-27 22:22:25,908 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 45.92 2021-04-27 22:22:30,992 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 45.92 2021-04-27 22:22:36,076 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 45.92 2021-04-27 22:22:41,153 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 46.71 2021-04-27 22:22:46,220 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 47.11 2021-04-27 22:22:51,352 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 47.11 2021-04-27 22:22:56,437 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 47.11 2021-04-27 22:23:01,505 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 47.89 2021-04-27 22:23:06,585 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 48.29 2021-04-27 22:23:11,663 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 48.29 2021-04-27 22:23:16,733 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 49.47 2021-04-27 22:23:21,825 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 49.47 2021-04-27 22:23:26,898 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 49.47 2021-04-27 22:23:31,973 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.26 2021-04-27 22:23:37,046 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.66 2021-04-27 22:23:42,142 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.66 2021-04-27 22:23:47,220 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.66 2021-04-27 22:23:52,303 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 50.66 2021-04-27 22:23:57,372 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 51.84 2021-04-27 22:24:02,457 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 51.84 2021-04-27 22:24:07,536 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 53.03 2021-04-27 22:24:12,630 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 53.03 2021-04-27 22:24:17,711 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 53.03 2021-04-27 22:24:22,782 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 53.82 2021-04-27 22:24:27,854 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 54.21 2021-04-27 22:24:32,931 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 55.0 2021-04-27 22:24:37,990 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:24:43,061 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:24:48,130 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:24:58,217 | test | INFO | pool-2-thread-4 | [task:check:355] Rebalance - status: none, progress: 100 Got this failure java.net.ConnectException: Connection refused: /172.23.136.107:8091 during connect (<_realsocket at 0xb type=client open_count=1 channel=[id: 0xb797212d, 0.0.0.0/0.0.0.0:32842] timeout=300.0>) 2021-04-27 22:24:58,263 | test | ERROR | pool-2-thread-4 | [rest_client:_http_request:752] Socket error while connecting to http://172.23.136.107:8091/nodes/self error [Errno 111] Connection refused 2021-04-27 22:25:00,319 | test | INFO | pool-2-thread-4 | [task:check:414] Rebalance completed with progress: 100% in 240.881999969 sec 2021-04-27 22:25:00,321 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:25:00,453 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:25:06,480 | test | INFO | pool-2-thread-5 | [table_view:display:72] Rebalance Overview +----------------+----------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+----------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 12.1623986467 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 95.1498610311 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 23.0068583905 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 93.7765110794 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 2.53747885434 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 6.61405643239 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 11.8101968366 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 92.2133333333 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 93.6207801732 | Cluster node | | 172.23.136.107 | ['n1ql,index'] | | | <--- IN --- | +----------------+----------------+-----------------------+---------------+--------------+ 2021-04-27 22:25:06,509 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:25:11,594 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:16,667 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:21,740 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:31,875 | test | INFO | pool-2-thread-5 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-27 22:25:31,920 | test | INFO | pool-2-thread-5 | [task:check:414] Rebalance completed with progress: 100% in 25.4409999847 sec 2021-04-27 22:25:31,921 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-27 22:25:32,055 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:25:32,244 | test | INFO | pool-2-thread-6 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 9.08984850252 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 94.1205438747 | Cluster node | | 172.23.136.107 | index, n1ql | 7.0.0-5017-enterprise | 3.72676510112 | --- OUT ---> | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 21.6266666667 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 93.8518716043 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 5.28276413821 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 5.96171634764 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 10.0650838757 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 91.0976361425 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 95.0790846514 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:25:32,279 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:25:37,381 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:42,444 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:47,515 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 65.0 2021-04-27 22:25:52,581 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: running, progress: 75.0 2021-04-27 22:26:02,673 | test | INFO | pool-2-thread-6 | [task:check:355] Rebalance - status: none, progress: 100 2021-04-27 22:26:03,651 | test | INFO | pool-2-thread-6 | [task:check:414] Rebalance completed with progress: 100% in 31.4099998474 sec 2021-04-27 22:26:03,654 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:26:03,799 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:26:14,211 | test | INFO | pool-2-thread-7 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 14.5080915318 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 95.3309016852 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 23.905 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 94.7126321842 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 1.21166666667 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 8.803946568 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 13.9064348928 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 92.4732706106 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 94.7908767427 | Cluster node | | 172.23.136.107 | ['kv'] | | | <--- IN --- | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:26:14,247 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:26:19,336 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:26:24,426 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 2.76 2021-04-27 22:26:29,516 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 6.9 2021-04-27 22:26:34,621 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 10.42 2021-04-27 22:26:39,710 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 13.58 2021-04-27 22:26:44,796 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 17.22 2021-04-27 22:26:49,878 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 20.34 2021-04-27 22:26:54,950 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 23.3 2021-04-27 22:27:00,042 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 26.54 2021-04-27 22:27:05,134 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 29.35 2021-04-27 22:27:10,220 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 32.17 2021-04-27 22:27:15,313 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 34.98 2021-04-27 22:27:20,413 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 37.24 2021-04-27 22:27:25,497 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 39.44 2021-04-27 22:27:30,601 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 41.85 2021-04-27 22:27:35,680 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 44.61 2021-04-27 22:27:40,801 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 46.34 2021-04-27 22:27:45,898 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 48.17 2021-04-27 22:27:50,976 | test | INFO | pool-2-thread-7 | [task:check:355] Rebalance - status: running, progress: 49.08 2021-04-27 22:27:56,046 | test | ERROR | pool-2-thread-7 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'15ac6a414b71a750fe3a3a2cfd8fa08c', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=fb577bb005500889a912a11e0e4ec443', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:27:56,842 | test | INFO | pool-2-thread-7 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:27:56,844 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587675778L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:55.778Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.20190.18>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a2_users_search:2021-04-27 22:27:20.2836704 -0700 PDT m=+2452.559014101]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = e197b89281485206bb3f29fba4e1f1ca'} 2021-04-27 22:27:56,845 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587580848L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:20.848Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:27:56,845 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619587576585L, u'shortText': u'node up', u'serverTime': u'2021-04-27T22:26:16.585Z', u'text': u"Node 'ns_1@172.23.138.127' saw that node 'ns_1@172.23.136.107' came up. Tags: [] (repeated 1 times, last seen 10.681331 secs ago)"} 2021-04-27 22:27:56,845 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.136.115', u'tstamp': 1619587575988L, u'shortText': u'node up', u'serverTime': u'2021-04-27T22:26:15.988Z', u'text': u"Node 'ns_1@172.23.136.115' saw that node 'ns_1@172.23.136.107' came up. Tags: [] (repeated 1 times, last seen 9.564689 secs ago)"} 2021-04-27 22:27:56,845 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.107', u'tstamp': 1619587574986L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.986Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.107\' in 0 seconds.'} 2021-04-27 22:27:56,846 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587574883L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.883Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:27:56,846 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587574330L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.330Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113',\n 'ns_1@172.23.136.110','ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = e197b89281485206bb3f29fba4e1f1ca"} 2021-04-27 22:27:56,848 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'memcached_config_mgr', u'type': u'info', u'node': u'ns_1@172.23.136.107', u'tstamp': 1619587574206L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.206Z', u'text': u'Hot-reloaded memcached.json for config change of the following keys: [<<"scramsha_fallback_salt">>]'} 2021-04-27 22:27:56,848 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 3, u'module': u'ns_cluster', u'type': u'info', u'node': u'ns_1@172.23.136.107', u'tstamp': 1619587574022L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.022Z', u'text': u'Node ns_1@172.23.136.107 joined cluster'} 2021-04-27 22:27:56,848 | test | ERROR | pool-2-thread-7 | [rest_client:print_UI_logs:2613] {u'code': 1, u'module': u'menelaus_web_sup', u'type': u'info', u'node': u'ns_1@172.23.136.107', u'tstamp': 1619587573954L, u'shortText': u'web start ok', u'serverTime': u'2021-04-27T22:26:13.954Z', u'text': u'Couchbase Server has started on web port 8091 on node \'ns_1@172.23.136.107\'. Version: "7.0.0-5017-enterprise".'} 2021-04-27 22:27:56,849 | test | ERROR | pool-2-thread-7 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'15ac6a414b71a750fe3a3a2cfd8fa08c', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=fb577bb005500889a912a11e0e4ec443', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:27:56,851 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-27 22:27:57,045 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:27:57,232 | test | INFO | pool-2-thread-8 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 10.1580793813 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 91.1717402355 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 91.6009733171 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 26.7558918631 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 93.8284876211 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 1.18604457057 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 30.6518007433 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 11.1974066883 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 87.1104036801 | --- OUT ---> | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 91.8209014925 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:27:57,263 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:28:02,378 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 2.96 2021-04-27 22:28:07,446 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 9.22 2021-04-27 22:28:12,526 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 17.3 2021-04-27 22:28:17,598 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 24.21 2021-04-27 22:28:22,661 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 31.41 2021-04-27 22:28:27,727 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 38.0 2021-04-27 22:28:32,792 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 43.82 2021-04-27 22:28:37,865 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 49.09 2021-04-27 22:28:42,930 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-27 22:28:47,997 | test | INFO | pool-2-thread-8 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-27 22:28:53,065 | test | ERROR | pool-2-thread-8 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'82ad4d5e29eff8659d296158acbf80de', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=0c28e1f260c205d6d6a35f7d6488b2fe', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:28:53,552 | test | INFO | pool-2-thread-8 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:28:53,552 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587731714L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:51.714Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.15379.19>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a5_users_search:2021-04-27 22:28:39.9396245 -0700 PDT m=+2532.217998501]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 7dbff9f8f30a4205c3cbee2787b91118'} 2021-04-27 22:28:53,553 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587729680L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:49.680Z', u'text': u'Shutting down bucket "travel-sample" on \'ns_1@172.23.136.110\' for deletion'} 2021-04-27 22:28:53,553 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587678064L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:58.064Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:28:53,555 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587677784L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:57.784Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:28:53,555 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587677342L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:57.342Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113',\n 'ns_1@172.23.136.105'], EjectNodes = ['ns_1@172.23.136.110'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 7dbff9f8f30a4205c3cbee2787b91118"} 2021-04-27 22:28:53,555 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587675778L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:55.778Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.20190.18>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a2_users_search:2021-04-27 22:27:20.2836704 -0700 PDT m=+2452.559014101]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = e197b89281485206bb3f29fba4e1f1ca'} 2021-04-27 22:28:53,555 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587580848L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:20.848Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:28:53,555 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.138.127', u'tstamp': 1619587576585L, u'shortText': u'node up', u'serverTime': u'2021-04-27T22:26:16.585Z', u'text': u"Node 'ns_1@172.23.138.127' saw that node 'ns_1@172.23.136.107' came up. Tags: [] (repeated 1 times, last seen 10.681331 secs ago)"} 2021-04-27 22:28:53,556 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 4, u'module': u'ns_node_disco', u'type': u'info', u'node': u'ns_1@172.23.136.115', u'tstamp': 1619587575988L, u'shortText': u'node up', u'serverTime': u'2021-04-27T22:26:15.988Z', u'text': u"Node 'ns_1@172.23.136.115' saw that node 'ns_1@172.23.136.107' came up. Tags: [] (repeated 1 times, last seen 9.564689 secs ago)"} 2021-04-27 22:28:53,556 | test | ERROR | pool-2-thread-8 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.107', u'tstamp': 1619587574986L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:26:14.986Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.107\' in 0 seconds.'} 2021-04-27 22:28:53,558 | test | ERROR | pool-2-thread-8 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'82ad4d5e29eff8659d296158acbf80de', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=0c28e1f260c205d6d6a35f7d6488b2fe', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:28:53,559 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:28:53,687 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:28:53,848 | test | INFO | pool-2-thread-9 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 12.4495850138 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 85.0522508708 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 79.3486503829 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 10.4320653306 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 82.9694505092 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 9.62333333333 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 53.8675644594 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 6.29406176564 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 29.5702464187 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 85.5867867768 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:28:53,868 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:28:58,937 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 2.29 2021-04-27 22:29:04,012 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 6.73 2021-04-27 22:29:09,089 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 10.73 2021-04-27 22:29:14,157 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 15.81 2021-04-27 22:29:19,223 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 20.56 2021-04-27 22:29:24,286 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 25.37 2021-04-27 22:29:29,351 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 31.28 2021-04-27 22:29:34,421 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 38.05 2021-04-27 22:29:39,535 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 44.82 2021-04-27 22:29:44,594 | test | INFO | pool-2-thread-9 | [task:check:355] Rebalance - status: running, progress: 47.93 2021-04-27 22:29:49,653 | test | ERROR | pool-2-thread-9 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'df499c6fc4b89bd0b116bc1d507de5ed', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=cba031d54b6b0c6e5931508040fe3406', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:29:49,901 | test | INFO | pool-2-thread-9 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:29:49,901 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587789110L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:29:49.110Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.7267.20>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a5_users_search:2021-04-27 22:28:39.9396245 -0700 PDT m=+2532.217998501]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 15683a3c5fbc53c9b23c8e4f89e6121f'} 2021-04-27 22:29:49,903 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587737807L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:57.807Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance (repeated 1 times, last seen 2.120685 secs ago)'} 2021-04-27 22:29:49,903 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587737807L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:57.807Z', u'text': u'Started rebalancing bucket travel-sample (repeated 1 times, last seen 3.372685 secs ago)'} 2021-04-27 22:29:49,904 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587734505L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:54.505Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.110\' in 0 seconds.'} 2021-04-27 22:29:49,904 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587734355L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:54.355Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-27 22:29:49,904 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587733965L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:53.965Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113',\n 'ns_1@172.23.136.110','ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 15683a3c5fbc53c9b23c8e4f89e6121f"} 2021-04-27 22:29:49,904 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587731714L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:51.714Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.15379.19>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a5_users_search:2021-04-27 22:28:39.9396245 -0700 PDT m=+2532.217998501]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 7dbff9f8f30a4205c3cbee2787b91118'} 2021-04-27 22:29:49,905 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587729680L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:49.680Z', u'text': u'Shutting down bucket "travel-sample" on \'ns_1@172.23.136.110\' for deletion'} 2021-04-27 22:29:49,905 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587678064L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:58.064Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:29:49,905 | test | ERROR | pool-2-thread-9 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587677784L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:27:57.784Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:29:49,907 | test | ERROR | pool-2-thread-9 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'df499c6fc4b89bd0b116bc1d507de5ed', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=cba031d54b6b0c6e5931508040fe3406', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:29:49,908 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-27 22:29:50,032 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:29:50,194 | test | INFO | pool-2-thread-10 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 12.6029200973 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 12.9952165869 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 12.381460309 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 5.41513537838 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 12.4368781561 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 2.96828386194 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 8.13986433559 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 10.8088964443 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 48.4245176882 | --- OUT ---> | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 15.6103902598 | --- OUT ---> | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:29:50,226 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:29:55,305 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 3.08 2021-04-27 22:30:00,391 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 7.51 2021-04-27 22:30:05,480 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 12.38 2021-04-27 22:30:10,571 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 16.42 2021-04-27 22:30:15,648 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 20.69 2021-04-27 22:30:20,716 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 24.77 2021-04-27 22:30:25,799 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 29.11 2021-04-27 22:30:30,874 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 33.4 2021-04-27 22:30:35,941 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 38.88 2021-04-27 22:30:41,015 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 43.51 2021-04-27 22:30:46,082 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 46.7 2021-04-27 22:30:51,147 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 49.24 2021-04-27 22:30:56,209 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-27 22:31:01,273 | test | INFO | pool-2-thread-10 | [task:check:355] Rebalance - status: running, progress: 50.0 2021-04-27 22:31:06,331 | test | ERROR | pool-2-thread-10 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a1a34cea3dc66facfd5ecfdec75e5c4e', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7d4c8ddd5768ea62836be2c11fae35b8', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:31:06,598 | test | INFO | pool-2-thread-10 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.106: 2021-04-27 22:31:06,598 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587866010L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:06.010Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.18745.21>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a3_users_search:2021-04-27 22:30:20.5472091 -0700 PDT m=+2632.828999101]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 98b70e0d1206bdc75546d3cfdd9d0eb8'} 2021-04-27 22:31:06,598 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587864242L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:04.242Z', u'text': u'Shutting down bucket "travel-sample" on \'ns_1@172.23.136.105\' for deletion'} 2021-04-27 22:31:06,598 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587864235L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:04.235Z', u'text': u'Shutting down bucket "travel-sample" on \'ns_1@172.23.136.110\' for deletion'} 2021-04-27 22:31:06,599 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587790845L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:29:50.845Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:31:06,599 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587790639L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:29:50.639Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:31:06,601 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587790314L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:29:50.314Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113'], EjectNodes = ['ns_1@172.23.136.110',\n 'ns_1@172.23.136.105'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 98b70e0d1206bdc75546d3cfdd9d0eb8"} 2021-04-27 22:31:06,601 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587789110L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:29:49.110Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.7267.20>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a5_users_search:2021-04-27 22:28:39.9396245 -0700 PDT m=+2532.217998501]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 15683a3c5fbc53c9b23c8e4f89e6121f'} 2021-04-27 22:31:06,601 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587737807L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:57.807Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance (repeated 1 times, last seen 2.120685 secs ago)'} 2021-04-27 22:31:06,601 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587737807L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:57.807Z', u'text': u'Started rebalancing bucket travel-sample (repeated 1 times, last seen 3.372685 secs ago)'} 2021-04-27 22:31:06,601 | test | ERROR | pool-2-thread-10 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587734505L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:28:54.505Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.110\' in 0 seconds.'} 2021-04-27 22:31:06,602 | test | ERROR | pool-2-thread-10 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'a1a34cea3dc66facfd5ecfdec75e5c4e', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=7d4c8ddd5768ea62836be2c11fae35b8', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:31:16,701 | test | INFO | MainThread | [travel_sample_app:run_app:102] #### Iteration :: 2 #### 2021-04-27 22:31:16,703 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:31:16,888 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:31:17,089 | test | INFO | pool-2-thread-11 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 4.53159114015 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 23.3991666667 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 27.1231073074 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 8.85340711173 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 26.092101535 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 11.3100636519 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 15.1447475875 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 4.2974641878 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 12.0951007925 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 12.9046773831 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:31:17,119 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:31:22,213 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 2.63 2021-04-27 22:31:27,308 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 7.12 2021-04-27 22:31:32,405 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 11.23 2021-04-27 22:31:37,494 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 15.72 2021-04-27 22:31:42,595 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 21.03 2021-04-27 22:31:47,673 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 25.75 2021-04-27 22:31:52,743 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 30.19 2021-04-27 22:31:57,842 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 35.16 2021-04-27 22:32:02,904 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 39.36 2021-04-27 22:32:07,967 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 42.71 2021-04-27 22:32:13,051 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 44.23 2021-04-27 22:32:18,108 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 46.02 2021-04-27 22:32:23,174 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 47.66 2021-04-27 22:32:28,240 | test | INFO | pool-2-thread-11 | [task:check:355] Rebalance - status: running, progress: 48.67 2021-04-27 22:32:33,302 | test | ERROR | pool-2-thread-11 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f847beeb1eba0db36d7a4a8c91570c8f', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=16db111338b15f247da1ca5b114a95d5', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:33,881 | test | INFO | pool-2-thread-11 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:32:33,882 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587952125L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:32.125Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.2667.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = a6320fd33bc212d51c313f9b6cd761ee'} 2021-04-27 22:32:33,884 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587878843L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:18.843Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:32:33,884 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877706L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.706Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.105\' in 0 seconds.'} 2021-04-27 22:32:33,884 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587877697L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.697Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.110\' in 0 seconds.'} 2021-04-27 22:32:33,884 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877620L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.620Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:32:33,884 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877559L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.559Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-27 22:32:33,885 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587877553L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.553Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-27 22:32:33,885 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877195L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.195Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113',\n 'ns_1@172.23.136.110','ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = a6320fd33bc212d51c313f9b6cd761ee"} 2021-04-27 22:32:33,887 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587866010L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:06.010Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.18745.21>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a3_users_search:2021-04-27 22:30:20.5472091 -0700 PDT m=+2632.828999101]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 98b70e0d1206bdc75546d3cfdd9d0eb8'} 2021-04-27 22:32:33,887 | test | ERROR | pool-2-thread-11 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587864242L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:04.242Z', u'text': u'Shutting down bucket "travel-sample" on \'ns_1@172.23.136.105\' for deletion'} 2021-04-27 22:32:33,888 | test | ERROR | pool-2-thread-11 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'f847beeb1eba0db36d7a4a8c91570c8f', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=16db111338b15f247da1ca5b114a95d5', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:33,888 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_out 2021-04-27 22:32:34,072 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:32:34,257 | test | INFO | pool-2-thread-12 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 5.9482837643 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 29.153680772 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 27.8258333333 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 5.35424553813 | --- OUT ---> | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 34.5583333333 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 3.32895130961 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 8.85340711173 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 6.40583333333 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 40.3646788226 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 46.835 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:32:34,288 | test | INFO | pool-2-thread-12 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:32:39,361 | test | ERROR | pool-2-thread-12 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fb3d23a3c8fa8807903147d84e01f0dc', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e199fc232bec6316476119696de7fe55', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:39,703 | test | INFO | pool-2-thread-12 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:32:39,703 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587957645L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:37.645Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.3468.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 35dd2d9c26a910544f951763cdb4fd07'} 2021-04-27 22:32:39,703 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587955053L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:35.053Z', u'text': u'Bucket "travel-sample" rebalance appears to be swap rebalance'} 2021-04-27 22:32:39,704 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587954787L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:34.787Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:32:39,704 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587954366L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:34.366Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.136.108',\n 'ns_1@172.23.136.112','ns_1@172.23.136.115',\n 'ns_1@172.23.136.113','ns_1@172.23.136.110',\n 'ns_1@172.23.136.105'], EjectNodes = ['ns_1@172.23.138.127'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 35dd2d9c26a910544f951763cdb4fd07"} 2021-04-27 22:32:39,706 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587952125L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:32.125Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.2667.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = a6320fd33bc212d51c313f9b6cd761ee'} 2021-04-27 22:32:39,706 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587878843L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:18.843Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:32:39,707 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877706L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.706Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.105\' in 0 seconds.'} 2021-04-27 22:32:39,707 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587877697L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.697Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.110\' in 0 seconds.'} 2021-04-27 22:32:39,707 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877620L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.620Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:32:39,707 | test | ERROR | pool-2-thread-12 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_storage_conf', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877559L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.559Z', u'text': u'Deleting old data files of bucket "travel-sample"'} 2021-04-27 22:32:39,709 | test | ERROR | pool-2-thread-12 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'fb3d23a3c8fa8807903147d84e01f0dc', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=e199fc232bec6316476119696de7fe55', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:39,710 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_in 2021-04-27 22:32:39,838 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 2021-04-27 22:32:40,009 | test | INFO | pool-2-thread-13 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+----------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+----------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 6.66827776435 | Cluster node | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 29.153680772 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 27.8258333333 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 5.35424553813 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 36.8608333333 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 0.768326930609 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 8.85340711173 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 6.40583333333 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 37.4615634609 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 46.835 | Cluster node | +----------------+---------------+-----------------------+----------------+--------------+ 2021-04-27 22:32:40,028 | test | INFO | pool-2-thread-13 | [task:check:355] Rebalance - status: running, progress: 0.0 2021-04-27 22:32:45,138 | test | ERROR | pool-2-thread-13 | [rest_client:_rebalance_status_and_progress:1510] {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'51af10ed91dbcfaf8488f66b2a258628', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=6750be4bc04e7be33fd1a80add731146', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:45,584 | test | INFO | pool-2-thread-13 | [rest_client:print_UI_logs:2611] Latest logs from UI on 172.23.136.105: 2021-04-27 22:32:45,585 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587961874L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:41.874Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.4279.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 89f9efba6f11a29a63dbbc338406cfef'} 2021-04-27 22:32:45,585 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587960118L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:40.118Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.138.127',\n 'ns_1@172.23.136.108','ns_1@172.23.136.112',\n 'ns_1@172.23.136.115','ns_1@172.23.136.113',\n 'ns_1@172.23.136.110','ns_1@172.23.136.105'], EjectNodes = [], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 89f9efba6f11a29a63dbbc338406cfef"} 2021-04-27 22:32:45,585 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587957645L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:37.645Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.3468.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = 35dd2d9c26a910544f951763cdb4fd07'} 2021-04-27 22:32:45,586 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587955053L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:35.053Z', u'text': u'Bucket "travel-sample" rebalance appears to be swap rebalance'} 2021-04-27 22:32:45,586 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_rebalancer', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587954787L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:34.787Z', u'text': u'Started rebalancing bucket travel-sample'} 2021-04-27 22:32:45,586 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587954366L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:34.366Z', u'text': u"Starting rebalance, KeepNodes = ['ns_1@172.23.136.114','ns_1@172.23.136.106',\n 'ns_1@172.23.136.107','ns_1@172.23.136.108',\n 'ns_1@172.23.136.112','ns_1@172.23.136.115',\n 'ns_1@172.23.136.113','ns_1@172.23.136.110',\n 'ns_1@172.23.136.105'], EjectNodes = ['ns_1@172.23.138.127'], Failed over and being ejected nodes = []; no delta recovery nodes; Operation Id = 35dd2d9c26a910544f951763cdb4fd07"} 2021-04-27 22:32:45,588 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_orchestrator', u'type': u'critical', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587952125L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:32:32.125Z', u'text': u'Rebalance exited with reason {service_rebalance_failed,eventing,\n {worker_died,\n {\'EXIT\',<0.2667.23>,\n {{badmatch,\n {error,\n {bad_nodes,eventing,prepare_rebalance,\n [{\'ns_1@172.23.136.115\',\n {error,\n {unknown_error,\n <<"Some apps are deploying or resuming on nodeId: c2e16dfe88967da8a18a2a76462c6b93 Apps: map[a4_users_search:2021-04-27 22:31:47.2669189 -0700 PDT m=+2719.551003901]">>}}}]}}},\n [{service_rebalancer,rebalance_worker,1,\n [{file,"src/service_rebalancer.erl"},\n {line,158}]},\n {proc_lib,init_p,3,\n [{file,"proc_lib.erl"},{line,234}]}]}}}}.\nRebalance Operation Id = a6320fd33bc212d51c313f9b6cd761ee'} 2021-04-27 22:32:45,588 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_vbucket_mover', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587878843L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:18.843Z', u'text': u'Bucket "travel-sample" rebalance does not seem to be swap rebalance'} 2021-04-27 22:32:45,588 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.105', u'tstamp': 1619587877706L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.706Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.105\' in 0 seconds.'} 2021-04-27 22:32:45,588 | test | ERROR | pool-2-thread-13 | [rest_client:print_UI_logs:2613] {u'code': 0, u'module': u'ns_memcached', u'type': u'info', u'node': u'ns_1@172.23.136.110', u'tstamp': 1619587877697L, u'shortText': u'message', u'serverTime': u'2021-04-27T22:31:17.697Z', u'text': u'Bucket "travel-sample" loaded on node \'ns_1@172.23.136.110\' in 0 seconds.'} 2021-04-27 22:32:45,589 | test | ERROR | pool-2-thread-13 | [task:call:267] Rebalance Failed: {u'errorMessage': u'Rebalance failed. See logs for detailed reason. You can try again.', u'type': u'rebalance', u'masterRequestTimedOut': False, u'statusId': u'51af10ed91dbcfaf8488f66b2a258628', u'statusIsStale': False, u'lastReportURI': u'/logs/rebalanceReport?reportID=6750be4bc04e7be33fd1a80add731146', u'status': u'notRunning'} - rebalance failed 2021-04-27 22:32:45,589 | test | INFO | MainThread | [cluster:run:188] Running cluster scenario: rebalance_swap 2021-04-27 22:32:45,750 | test | INFO | MainThread | [cluster:update_nodes_in_cluster:40] Current master: 172.23.136.105 ERROR 2021-04-27 22:32:45,811 | test | INFO | MainThread | [app_basetest:_AppBase__print_step:90] ############################################################ 1. Closing SDK client connections ############################################################ 2021-04-27 22:32:45,914 | test | INFO | pool-2-thread-14 | [table_view:display:72] Rebalance Overview +----------------+---------------+-----------------------+---------------+--------------+ | Nodes | Services | Version | CPU | Status | +----------------+---------------+-----------------------+---------------+--------------+ | 172.23.136.114 | index, n1ql | 7.0.0-5017-enterprise | 6.66827776435 | --- OUT ---> | | 172.23.136.106 | kv | 7.0.0-5017-enterprise | 53.36 | Cluster node | | 172.23.136.107 | kv | 7.0.0-5017-enterprise | 55.72879774 | Cluster node | | 172.23.138.127 | cbas | 7.0.0-5017-enterprise | 15.5699667818 | Cluster node | | 172.23.136.108 | kv | 7.0.0-5017-enterprise | 62.4746872891 | Cluster node | | 172.23.136.112 | backup | 7.0.0-5017-enterprise | 4.90506197054 | Cluster node | | 172.23.136.115 | eventing, fts | 7.0.0-5017-enterprise | 53.1136202658 | Cluster node | | 172.23.136.113 | index, n1ql | 7.0.0-5017-enterprise | 16.0522017401 | Cluster node | | 172.23.136.110 | kv | 7.0.0-5017-enterprise | 53.9451315044 | Cluster node | | 172.23.136.105 | kv | 7.0.0-5017-enterprise | 61.3928217265 | Cluster node | +----------------+---------------+-----------------------+---------------+--------------+ 2021-04-27 22:32:45,934 | test | INFO | pool-2-thread-14 | [task:check:355] Rebalance - status: running, progress: 0.0 ====================================================================== ERROR: run_app (bucket_collections.travel_sample_app.TravelSampleApp) ---------------------------------------------------------------------- Traceback (most recent call last): File "pytests/bucket_collections/travel_sample_app.py", line 152, in run_app cluster_scenario.run(op_type, File "pytests/bucket_collections/app/scenarios/cluster.py", line 190, in run Cluster.scenarios["scenario_%s" % op_type](self, kwargs) File "pytests/bucket_collections/app/scenarios/cluster.py", line 139, in scenario_rebalance_swap self.cluster.master = self.spare_nodes[0] IndexError: index out of range: 0 ---------------------------------------------------------------------- Ran 1 test in 1469.208s During the test, Remote Connections: 31, Disconnections: 31 SDK Connections: 0, Disconnections: 0 FAILED (errors=1) summary so far suite bucket_collections.travel_sample_app.TravelSampleApp , pass 0 , fail 1 failures so far... bucket_collections.travel_sample_app.TravelSampleApp.run_app testrunner logs, diags and results are available under /data/workspace/temp_centos7_4node_jython/logs/testrunner-21-Apr-27_22-08-27/test_1 Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0. Use '--warning-mode all' to show the individual deprecation warnings. See https://docs.gradle.org/6.8/userguide/command_line_interface.html#sec:command_line_warnings BUILD SUCCESSFUL in 24m 51s 2 actionable tasks: 2 executed + '[' false == true ']' Archiving artifacts Recording test results Build step 'Publish JUnit test result report' changed build result to UNSTABLE [description-setter] Description set: 1.1.5 -P client_version=default' [BFA] Scanning build for known causes... [BFA] No failure causes found [BFA] Done. 0s Notifying upstream projects of job completion Email was triggered for: Unstable (Test Failures) Sending email for trigger: Unstable (Test Failures) An attempt to send an e-mail to empty list of recipients, ignored. Finished: UNSTABLE